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1.  Tutorial  Objectives 


The  purpose  of  this  tutorial  is  3-fold: 

•  Familiarize  participants  with  the  fundamental  principles  of  adaptive  Intelligent  Tutoring 
System  (ITS)  design. 

•  Illustrate  how  ITS  design  influences/enables  self-regulated  learning  (SRL). 

•  Discuss  the  need  for  standards  for  authoring  of  ITS,  modeling  of  learners  and  experts, 
automated  instructional  strategies,  and  methods  of  analysis  for  ITS  technologies. 


2.  Fundamentals  of  Tutoring 


This  section  reviews  the  fundamentals  of  ITSs.  Slide  3  illustrates  a  typical  tutor-user  interface 
used  by  the  ITS  to  deliver  content  and  feedback  to  the  learner,  and  receive  learner  input.  This 
interface  may  include  a  tutor  natural  language  feedback  window,  which  is  used  by  the  ITS  to 
provide  verbal  feedback  or  direction  through  a  virtual  human.  It  usually  provides  a  text  feedback 
window,  a  content  presentation  window,  and  a  learner  response  window  where  the  learner 
provides  text  input  in  response  to  ITS  directions,  questions,  or  feedback.  A  running  log  of  the 
conversation  or  chat  window  may  also  be  part  of  the  tutor-user  interface. 


0  RDEcoi^  Typical  Tutor-User  Interface  AKL 
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Slide  4  illustrates  the  “sweet  spot”  for  ITSs,  the  area  where  they  may  be  most  useful.  This  is 
within  tutoring  for  complex  skills  versus  simpler  tasks.  To  support  SRL,  tutors  adapt  their 
delivery  and  the  challenge  level  of  scenarios  to  match  those  of  the  learner.  If  there  is  no  adaptive 
change  to  match  the  learner’s  needs,  then  this  is  simple  computer-based  training  where  the 
training  is  the  same  for  everyone  and  not  adaptive  tutoring. 

What  is  SRL?  SRL  is  learning  (acquisition  of  knowledge  or  skills)  that  is  guided  by 
metacognition  (thinking  about  one’s  thinking),  strategic  action  (planning,  monitoring,  and 
evaluating  personal  progress  against  a  standard),  and  motivation  to  learn.  It  takes  great 
awareness  and  discipline  to  guide  your  own  learning.  Expert  human  tutors  are  much  more 
effective  and  efficient  at  guiding  learning.  Just  as  expert  human  tutors  guide  learners,  adaptive 
ITSs  may  also  augment  SRL  by  shaping  instructional  content  and  scaffolding  support  to  meet  the 
learner’s  needs.  The  computer-based  ITSs  must  be  situationally  aware  of  the  learner’s  state  and 
the  instructional  context  to  be  effective  guides  and  support  efficient  instruction. 


As  noted  in  Slide  4,  complex  skills  may  include  cognitive  tasks  where  challenging  decision¬ 
making  and  strategic  thinking  are  exercised  or  affective  tasks  where  interpersonal  skills  and 
ethical  conduct  are  tested  or  psychomotor  skills  where  coordination  and  timing  are  critical  to 
physical  tasks  (e.g.,  land  navigation)  or  operating  sophisticated  systems.  ITSs  may  be  used  to 
prepare  for  live  training,  enhance  learning  in  conjunction  with  virtual  training  environments,  or 
act  as  job  or  decision  aids  during  actual  operations. 
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-  Cognitive  (ie.g.  complex  decision-making,  strategic  thinking) 
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When  and  Where 
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The  primary  reasons  that  tutors  are  limited  in  their  use  in  education  and  training  (see  Slide  5)  are 
1)  they  have  insufficient  ability  to  adapt  to  learner  needs  and  2)  they  are  expensive  to  author.  It 
takes  approximately  200  h  of  interdisciplinary  team  labor  to  make  1  h  of  coursework  for  tutoring. 
More  adaptive  equals  more  expensive. 
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Limitations  of  Current  ITS 
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Expensive  to  author  and  are  insufficiently  adaptive*  to  support  highly 
effective  and  tailored  training  experiences  across  a  broad  spectrum 
of  military  tasks. 


•  Adaptiveness  drives  the  need  for  additional 
authoring...  more  authoring,  more 
development  time,  more  cost 

•  Adaptiveness  is  largely  based  on  knowledge 
of  learner  performance...  other  attributes 
influence  learning  (e.g.,  individual 
differences) 


•  ITSs  have  been  primarily  applied  in  limited, 
well-defined  domains...  with  limited 
application  to  military  tasks 


'adaptive  systems  customize  themselves  automatically  in  response  to  users  or  changes  in 
the  environment. 
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Significant  evidence  exists  that  computer-based  intelligent  tutoring  systems  are  just  as  effective 
as  human  tutors  under  certain  conditions  and  generally  within  more  well-defined  domains  (e.g., 
mathematies,  physies)  where  there  is  generally  one  correet  answer  to  a  problem.  The  goal  is  for 
ITSs  to  be  as  effective  as  expert  human  tutors  under  all  conditions  and  domains.  The  effect  sizes 
shown  in  Slide  6  are  baselined  against  traditional  classroom  training.  The  ultimate  goal  for  ITSs 
is  to  impaet  learning  with  effeet  sizes  equivalent  to  raising  average  (“C”)  students  to  experts 
(“A”  students)  through  tailored  instruction  and  reinforcement  of  deep  learning  principles.  In 
other  words,  the  goal  is  to  be  as  effective  as  or  more  effective  than  expert  human  tutors. 
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Tutoring  Methods  and  Effect  Sizes... 

2.00  Skilled  human  tutors  (Bloom,  1984)  ( |  score  from  50th  to  98th) 

1.05  Other  tutoring  systems  ( t  median  score  from  50th  to  85th) 

PACT  Geometry  Tutor  (Anderson,  Corbett,  Koedinger  &  Pelletier,  1995) 
Atlas-Andes  (VanLehn,  et  al.,  2005;  Rose,  et  al,  2001) 

Diagnoser  -  physics  (Hunt  &  Minstrell,  1994) 

Sherlock  (Lesgold,  et  al.,  1988) 

0.80  AutoTutor  (20  experiments)  (Graesser,  et  al,  2001-present ) 

0.79  Skilled  human  tutors  (VanLehn,  2011) 

( t  median  score  from  50‘**  percentile  to  79*^**) 

0.42  Unskilled  human  tutors  (Cohen,  Kulik,  &  Kulik,  1982) 

( t  median  score  from  50^**  percentile  to  66“*  percentile) 

0.00  Baseline  -  traditional  classroom  training 


(^reshoid^ 


Adapted  from  information  from  Dr.  A 
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Graesser,  University  of  Memphis,  and  Dr.  Beveriy  Wooif,  University  of  Massachusetts  -  Amherst. 
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Many  organizations  are  feeling  the  crunch  to  achieve  more  with  less.  While  training  and 
education  remain  important  to  maintaining  organizational  competency,  it  may  not  be  practical  or 
feasible  to  achieve  and  maintain  competency  in  a  traditional  classroom.  One-to-one  training 
(Bloom  1984)  has  been  shown  (Slide  7)  to  be  more  effective  than  classroom  training,  but  it  is  not 
practical  to  have  one-to-one  tutoring  for  every  person  in  a  large  organization. 


0  RDECon^^  Need  and  S&T  Opportunity 
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Need:  A  smaller  Force  requires  each  Warfighter  to  have  expertise  for  a  greater 
range  of  skills  for  complex  missions.  Need  to  achieve  expertise  faster  with  fewer 
resources. 


Opportunity:  Accelerate  development  of  expertise  by  developing  intelligent 
tutoring  systems  as  effective  as  human  tutors  -  with  cost-effective  features. 


Classroom  Training  vs.  1:1  Live  Instruction 


Learning  Improvements 


-  Bloom,  B.S.(1984) 


Shift  from  one-size-fits-all  training  ^  affordable,  personalized  learning. 

Enhances  mission  readiness.  Reduces  training  cost. 


PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


So,  if  we  wanted  an  ideal  tutoring  system  that  could  adapt  to  our  needs,  what  would  it  look  like? 
Slide  8  shows  a  set  of  salient  charaeteristies  for  a  platinum-level  tutor  as  deseribed  by  Sottilare 
and  Gilbert  (2011). 
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Salient  Characteristics  of  an  ftjjj 
Ideal  Tutoring  System  AAixL 


»  self-regulated  -  support  learning  of  individuals  and  teams  (Army  requirement) 

•  adaptive  -  use  Al  to  tailor  instruction  to  the  learning  needs  of  individuals  and  teams  of 
Soldiers 

»  effective  &  credible  -  as  good  or  better  than  an  expert  human  tutor 

•  relevant  -  support  military  training  in  both  ill-defined  and  well-defined  environments 
»  accurate  &  valid  -  use  optimal  instructional  methods  based  on  empirical  results 

•  usable  -  tailored  to  different  users  (trainees,  trainers,  developers,  designers...) 

»  accessible  -  service-oriented,  available  anywhere  24/7/365 

•  affordable  -  easy  to  author,  promotes  standards  and  reuse 

»  persistent  -  models  the  learning  needs  of  Soldiers  across  their  careers 


Sottilare,  R.  and  Gilbert,  S.  (2011).  Considerations  for  tutoring,  cognitive  modeling,  authoring  and  interaction  design  in  serious 
games.  Authoring  Simulation  and  Game-based  Intelligent  Tutoring  workshop  at  the  Artificial  Intelligence  in  Education 
Conference  (AIED)  2011 ,  Auckland,  New  Zealand,  June  2011. 
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Before  we  continue  in  our  discussion  of  tutoring  systems,  we  should  examine  their  fundamental 
elements  and  processes  (Slide  9).  Nearly  every  tutoring  system  has  4  fundamental  elements:  a 
learner  model,  a  pedagogical  (instructional)  model,  a  domain  model,  and  a  communication 
model.  The  green  boxes  in  the  slide  below  show  these  fundamental  elements  as  modules  versus 
models  because  they  manage  processes  in  addition  to  modeling  the  learner,  the  instruction,  the 
domain,  and  the  communication. 


Domain 
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ITSs  use  the  processes  in  Slide  9  to  assess  the  learner  and  manage  instruction.  ITSs  are  defined 
as  intelligent  software-based  agents  that  guide  instruction  by  observing  and  interpreting  learner 
data  (behaviors,  physiology,  demographics)  to  classify  learner  states  (e.g.,  engagement, 
competency,  emotions).  The  ITS  uses  these  states  to  adapt/tailor  instructions  to  match  the 
learner’s  capabilities  and  needs  to  optimize  learning.  Slide  10  shows  the  interaction  between 
tutoring  agents,  the  learner,  and  the  training  environment  (e.g.,  simulation,  game,  presentation). 
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Intelligent  computer-guided  instruction  where  software- 
based  agents: 

•  observe  and  interpret  learner  data  to  determine  learner  states 

•  adapt  and  tailor  instruction  to  match  the  learner’s  capabilities  and 
needs  to  optimize  effective  problem  solving  and  decision-making. 
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Finally,  the  adaptive  tutoring  learning  effect  model  shown  in  Slide  1 1  illustrates  the  interactive 
loops  of  the  tutoring  process.  Learner  module  data/processes  are  shown  in  green  boxes. 
Pedagogical  module  data/process  are  shown  in  light  blue  boxes  and  domain  module  processes  in 
light  orange. 
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Updated  Adaptive  Tutoring 
Learning  Effect  Modei 
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3.  Generalized  Intelligent  Framework  for  Ttitoring 


Traditionally,  ITSs  are  expensive  and  time  consuming  to  develop.  Additionally,  they  tend  to  be 
linked  to  specific  content  and  are  not  easily  changed.  Reuse  in  tutoring  systems  is  virtually 
nonexistent.  The  Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  is  an  open-source, 
domain-independent  intelligent  tutoring  framework  (Slide  13).  It  is  intended  to  provide 
flexibility  and  the  ability  to  create  full  tutors  with  content  of  the  author’s  choice.  By  being 
domain-independent,  it  allows  for  the  reuse  of  materials  and  a  reduction  in  both  the  time  and 
monetary  costs  of  developing  tutoring  systems. 
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An  S&T  effort  to  develop  an  open-source  tutoring  architecture  to: 

-  capture  best  tutoring  practices  and  support  rapid  authoring,  reuse  and 
interoperability  of  ITSs 

-  lower  costs  and  entry  skills  needed  to  author  ITSs 

-  enhance  the  adaptiveness  of  ITSs  to  support  self-regulated  learning  (SRL) 
per  the  Army  Learning  Model 
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•  tools 

•  methods 
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The  fundamental  tutoring  processes  within  GIFT  are  consistent  with  the  previously  described 
components  of  the  tutoring  process,  which  are  reviewed  in  Slide  14. 
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As  with  most  ITSs,  GIFT  has  4  primary  modular  components:  Domain  Module,  Learner  Module, 
Pedagogical  Module,  and  Gateway  Module.  Additionally,  this  diagram  represents  the  process 
that  occurs  with  user  input  to  GIFT  when  the  user  is  interacting  with  a  training  application. 
Additional  information  about  these  modules  and  their  processes  is  provided  further  along  in  the 
“GIFT  Tutors  GIFT”  portion  of  this  tutorial.  In  addition  to  the  4  primary  modules,  GIFT  also 
provides  a  sensor  module  that  is  used  to  capture  behavioral  and  physiological  data  about  the 
learner  which  can  then  be  used  to  interpret/classify  that  data  into  learner  states  for  use  by  the 
learner  module  and  the  pedagogical  module  as  described  in  the  “Adaptive  Tutoring  Learning 
Effect  Chain”.  The  interaction  of  the  modules  is  illustrated  in  Slide  15. 
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The  modules  in  GIFT  can  be  launched  manually  through  the  Module  Monitor  shown  in  Slide  16. 
The  image  on  the  left  is  the  Module  Monitor  before  the  modules  have  been  launched.  The  image 
on  the  right  is  the  Module  Monitor  after  the  modules  have  been  successfully  launched.  In  GIFT 
3.0  and  above,  it  is  also  possible  to  launch  the  modules  in  one  step,  which  includes  the  launching 
of  the  tutor  web  page. 


Slide  17  provides  a  visual  of  the  Admin  Tools  in  the  Module  Monitor  (left)  and  the  GIFT  login 
screen  on  the  tutor  web  page  (right).  There  is  also  a  simple  login  page  that  can  be  used  for 
experiments  and  does  not  require  a  password. 
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P  GIFT  Monitor  Module 


ks  LwebcanB^I  Systeni  Msgs  |_Sensof,|  Learner  State] 

,  Module  Contr^  Admin  Tools  |  Active  "nnnintT.  I 


Launch  Admin  Server  (GAS) 


Open  Survey  Authoring  System  (SAS) 


Open  Event  Report  Tool  (ERT) 


Open  Authoring  Tool  (GAT) 


Launch  Dl^  Authoring  Tool  (DAT) 


[  Latnch  Learner  Contig.  Authoring  Tool  (LCAT)  


Launch  Course  Authoring  Tool  (CAT) 


Launch  Pedagogy  Config.  Authoring  Tool  (PCAT)  J 


Launch  Import  Tool 


SFC  Paul  Ray  Smith  Simulation  &  Training  Technology  Center 


PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


Within  the  Admin  Tools  Tab  is  a  series  of  authoring  tools.  The  GAT  (GIFT  Authoring  Tool)  is  a 
new  element  of  GIFT  20I4-I-X,  as  is  shown  in  Slide  18.  It  provides  a  user-friendly  way  for 
authors  to  create  their  courses  in  GIFT.  The  course  selection  dashboard  is  pictured  in  the  current 
slide.  Each  line  represents  a  current  course  within  GIFT,  and  the  green  checkmark  indicates  that 
it  is  complete  and  validated.  Courses  can  be  opened  by  highlighting  the  name  and  then  clicking 
“Edit”. 


0  RDECOn^  GIFT  Course  Authoring  Tool 


Gift  Daihboard  • 


SFC  Paul  Ray  Smith  Simulation  &  Training  Technology  Center 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


The  functionality  of  the  new  GAT  is  demonstrated  in  Slide  19.  The  transitions  and  elements  of 
the  course  are  presented  on  the  left  side  of  the  screen.  Once  one  of  transitions  is  selected,  the 
associated  fields  to  enter  content  into  are  displayed  on  the  right  side  of  the  screen.  Additional 
transitions  can  be  added  using  the  menu  on  the  top  left  of  the  page. 
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W  ROECOM 


GIFT  Course  Authoring  Tool 


Gift  Course  Authoring  Tool  ® 


Course  TranMUon  Return  lo  Da 
Course  S  AOd  AAR 
Transilior  Present  Survey 
AOd  Oudaitca 

^  V  Md  Lesson  Material 

Add  Merrill  s  Brancft  Point 
®  '  ■>  Add  Training  App 

^  Delete 

9  null  j 


#  O  null 

fg  ■ 

MIDSURVEY 

g  e 

null 

a  . 

LPCONTENTQ 

<g  O  null 

APPUEDLPQ 

Course  Name. 
Logic  Puzzle  Tntorial 


insert  AAR 
Insert  Present  Survey 
Insert  Guidance 
Insert  Lesson  Material 
Insert  Mermrs  Brancii  Point 
Insert  Training  App 

Survey  Context: 

Logic  Puzzle  study 


SFC  Paul  Ray  Smith  Simulation  &  Training  Technology  Center 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


Surveys  are  authored  in  GIFT  through  the  Survey  Authoring  System.  It  provides  the  ability  to 
create  questions  and  surveys.  Additionally,  it  can  be  programmed  such  that  the  surveys  are 
automatically  graded. 


0  RDECOM 


Survey  Authoring  Tool  ARL 


GIFT  Survey  System 


Create  Question  Manage  Shared  Option  Lists  Manage  Categories 


Narrow  Results  Hover  over  the  question  to  get  more  delais 


Question  Type 

n 

QuesHon 

What  Is  your  age? 
FBInTheBt^ 

Fa  In  The  Btonk 

1 

EdH 

MuMpte  Choice  ' 

RadngScale 

2 

What  is  your  gender? 

Erki 

Matrix  Of  Choices 

Rating  Scale 

Category 

3 

What  is  your  rank? 

Edk 

CLS  Priority  Important 

CLS  Priority  Life-saving 

Fa  In  The  Blank 

What  is  your  Maiary 

CLS  Priority  Ufe-threaiening  ■  I 

Occupational  Spedaity 

ExpCcit  Feedback  Oemog 

(MOS)? 

HPFama»ity 

Fa  In  The  Biwik 

Immersive  TervlefKies  Que< 
Logic  Puzzle  Demographics  - 

5 

How  many  houre  of  sleep  did 
you  get  last  night? 

Edk 

Fa  In  The  Blank 

Have  you  had  any  caffeine  in  _ ^ 

6  the  last  two  hours?  [  EdM  ' 

Fa  In  The  Bi»ik 


Copy  [  Detete  j 

Copy  I  Detete  ] 

Copy  [  Detete  j 

Copy  [  Ootete  j 

Copy  [  Detete  j 

Copy  *  Delete  J 


SFC  Paul  Ray  Smith  Simulation  &  Training  Technology  Center 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


The  next  set  of  slides  (pp.  1 1-15)  provides  an  animated  sequence  that  explains  GIFT  in  simple 
terms.  GIFT  has  3  distinct  functions.  First,  it  is  an  authoring  capability  to  develop  new  ITS 
components  and  whole  tutoring  systems.  Second,  it’s  an  instructional  manager  that  integrates 
selected  tutoring  principles  and  strategies  for  use  in  ITSs.  Third,  it’s  an  experimental  test  bed  to 
analyze  the  effectiveness  and  impact  of  computer-based  tutoring  systems  components,  tools,  and 
methods.  GIFT  is  intended  to  be  a  community  platform  for  you  to  contribute  to  and  to  help  you 
with  your  research  and  development. 
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S®ini®imDk@dl  Dini&©DDOi®inifi  Fiiiiin]ii@w©irl!s  te  lHuiMinii 


GIFT  is...  an  Intelligent  Tutoring 
System 

...an  experimental  testbed 
...an  architecture 
...a  research  nexus 
...a  community  platform 


. . .  with  tools  for  authoring 

...  with  data  extraction/logging 

...  with  interchangeable  parts 

...  as  an  exit  vector  for  research 
projects 

...  informed  by  Advisory  Boards 


GIFT  is  based  on  a  learner-centric  approach  and  is  therefore  designed  to  be  consistent  with  the 
Adaptive  Tutoring  Learning  Effect  Chain.  In  this  chain,  the  learner  data  informs  learner  states, 
which  then  informs  instructional  strategy.  Selection  of  the  appropriate  strategy  at  the  correct  time 
is  expected  to  lead  to  learning  gains. 

Each  of  these  processes  is  captured  in  individual  modules  of  GIET.  Eor  this  tutorial,  we  will  go 
though  the  functions,  inputs,  and  outputs  of  each  GIFT  module.  Let’s  talk  about  the  first  one,  the 
Domain  Module.  Although  the  Domain  Module  is  one  module  in  reality,  it  is  divided  into  2 
modules  for  you  to  better  understand  each  of  its  2  functions:  performance  assessment  and 
strategy  implementation. 

The  learning  effect  chain  starts  by  monitoring  learner  performance  and  sensor  data.  Learner 
performance  is  assessed  from  data  captured  in  a  training  application,  which  is  fed  into  GIFT’S 
Domain  Module  via  the  Gateway  Module.  GIFT  processes  learner  interaction  data  by  comparing 
the  Domain  Module  inputs  against  designated  models  of  expert  performance.  In  order  to  keep 
GIFT  generic  to  all  training  applications,  domains  are  represented  as  a  hierarchical  structure  of 
“Concepts”. 


11 


•  This  tutorial  will  show  you  GIFT’S  Main  Modules  and  Processes. 


•  The  implementation  of  GIFT  is  centered  around  the  Adaptive  Tutoring  Learning 
Effect  Chain  [ATLEC).  Each  component  is  intended  to  inform  strategies  to  influence 
learning  gains. _ 


learner 

data 


mforms 


■> 


learner 

states 


instructional 

strategy 

selection 


instructional 

context 


instructional  mjences  ^ 
tactic  * 

selection 


iearning  _ 
gains 


For  all  identified  Concepts  and  Subconcepts,  a  learner  can  perform  “at  standard”,  “above 
standard”,  or  “below  standard”,  as  determined  by  comparsion  of  performance  to  the  expert 
model.  Outputs  from  the  Domain  Module  are  fed  into  the  Learner  Module  and  contain 
performance  states  associated  with  specific  concepts  represented  in  the  domain  representation. 


Sensor 


t _ 


] 

Domain 

Learner 

->■ 

Pedagogical 

(Strategy 

Implementation) 

Gateway 


What  Does  This  Module  Do? 

1.  Provides  Performance  Assessments  romcess 

shown*) 

2.  Implements  instructional  “Strategies”  as  “Tactics” 

GIFT  Supports  These  Training  Applications: 


Inputs: 

•Student 
Actions 
(as  Gateway 
messages) 


& 


Outputs: 

•Performance  Assessments  for 
each  Concept  and  Sub- 
Concept 

•Above-,  At-,  or  Below- 
Expectation  Measurements 
•Generic,  domain-independent 
message  structure - 


GIFT  also  uses  information  collected  from  sensors  to  monitor  a  learner’s  reactive  states  while 
interacting  with  a  system.  The  Sensor  Module  takes  in  raw  sensor  data  streams  as  inputs  and 
applies  filters  to  convert  the  data  into  metrics  correlated  with  cognitive  and  affective  states.  This 
filtered  data  is  passed  to  the  Learner  Module  for  inferring  affective  and  cognitive  state 
determinations.  You  wouldn’t  use  the  same  instruction  for  a  bored  student  as  an  energetic  one, 
would  you?  Sensors  may  be  used  to  collect  data  about  the  learner  and  determine  the  learner’s 
states.  Data  from  the  learner  may  be  filtered  or  unfiltered. 
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Gateway 


Dsoiigor  [l(6)(^[u][]© 


What  Does  This  Module  Do? 

1 .  Provides  filtered  sensor  informati^  ^  0 
to  help  assess  the  learner’s  cognitiV^^ 
and  affective  states 


3  states  \ 

^GIFT^BuppoVts: 


Training 

Application 


Inputs: 

Student  Sensor  Dat 
from  hardware 
and/or  software 
Sensors. 

(It’s  your  decision!) 


Outputs: 


Filtered  Sensor  Information 
based  on  configuration 


So  far  we’ve  learned  that  performance  information  from  the  Domain  Module  and  sensor 
information  from  the  Sensor  Module  serve  as  inputs  to  the  next  function  of  the  learning  effect 
chain,  the  Learner  Module.  The  Learner  Module  consumes  this  information  and  performs 
processes  to  designate  a  “Learner  State”,  which  may  be  cognitive  (e.g.,  engagement),  affective 
(e.g.,  emotions),  or  physical  (e.g.,  fatigue  or  arousal).  Once  a  state  is  classified,  the  results  are 
then  fed  to  the  Pedagogical  Module.  In  addition,  the  Learner  Module  maintains  information 
about  the  learner’s  stable  traits,  experiences,  and  competencies,  which  are  also  used  to  customize 
strategy  selections  for  the  individual  learner. 


^  Pedagogical 
Gateway  <— 


Domain 

(Strategy 

Implementation) 


Training 

Application 


What  Does  This  Module  Do? 

1 .  Provides  a  “Picture”  of  the  Learner 

Independent  of  training  application  or  sensors 


GIFT  Supports: 

•Current:  Changes  in  performance 


•Future:  Adaptive  Resonance  Theory  |  i 

More  research  is  needed  in  this 


Inputs: 

Filtered  Sensor  data 
Persistent  Learner  Data 
Performance  State 


Outputs: 

“Learner  State”  as  a  collection  of: 

•  Affective,  Cognitive,  and 
Performance  Dimensions 

•  Provides  Current,  Short-Term, 
and  Long-Term  views  of  state 


The  Pedagogical  Module  uses  the  “Picture  of  the  Learner”  state  information  and  learner  trait 
data,  and  then  recommends  generic  strategies  to  accommodate  individual  learners.  If  the  student 
is  bored  and  passing  your  course,  then  you  probably  want  to  adjust  the  course.  For  instance,  the 
Pedagogical  Module  may  provide  guidance  and  feedback  to  aid  in  performance,  adjust  course 
difficulty,  present  more  interactive  material,  or  request  an  assessment  of  the  learner’s  ability. 
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What  Does  This  Do? 


1 .  Provides  generic  strategies  ,  such  as: 
•  “Provide  Hint”  (Microadaptive) 


Inputs: 

“Sources  of  Adaptation” 
•Learner  Performance  State 
•Learner  Affective/Cognitive  State 
•Learner  Abilities 
•Learner  Traits 


•  “Give  Practice  Content” 
(Macroadaptive) 


GIFT  Supports: 

An  Engine  for  MacroAdaptive  Pedagogy 
(EM2AP) 


Outputs: 

“Generic  Instructional  Strategy 

Recommendation” 
•Provide  Guidance 
•Adapt  Problem/Scenario 
•Request  Assessment 


Once  the  Pedagogical  Module  uses  learner  state  information  to  select  a  strategy,  the  strategy 
selected  is  then  sent  to  the  Domain  Module.  The  Domain  Module  then  chooses  an  appropriate 
method  to  implement  the  request  as  a  defined  tactie  or  aetion.  The  seleetion  of  the  tactic  is  based 
on  the  learner’s  ability  levels.  For  example,  it  may  be  more  appropriate  for  one  learner  to  receive 
a  hint  while  a  metacognitive  prompt  may  be  more  suitable  for  another. 


Sensor 


Domain 

(Performance 

Data) 


Learner 


What  Does  This  Module  Do? 

1.  Provides  Performance  Assessments 

2.  Implements  instructional  “Strategies”  as 

“Tactics” _ 

GIFT  Supports: 

Virtual  Characters  (all  Domains) 
HTML  Items  (all  Domains) 
n-game  characters  (Virtual  Medic  Sim) 
n-game  events  (Virtual  BattleSpace  2) 


Training 

Application 


Inputs: 

“Instructional  Strategy 
Recommendation” 
•Provide  Guidance 
•Adapt  Problem/Scenario 
•Request  Assessment 


Outputs: 

•Change  in  user  experience 
(Dependent  on  Domain  Module 
Triggering  the  Effect  Chain) 
•In-game  feedback,  scenario 
changes,  reflective  prompts,  etc. 


The  Gateway  Module  receives  the  tactic  that  was  output  by  the  Domain  Module  and  presents  it 
to  the  learner.  Depending  on  the  type  of  request,  this  can  involve  actions  to  execute  tactics  within 
the  training  application  environment  (e.g.,  serious  game)  or  by  the  Tutor  User  Interface,  a 
window  to  present  information  to  and  receive  information  from  the  learner. 
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The  Gateway  Module  translates  this  interface  into  GIFT-assessable  messages,  allowing  for  the 
whole  learning  effect  to  operate.  If  you  already  have  a  training  application,  the  Gateway  Module 
is  where  you  can  build  an  interface  to  it  and  add  individualized  and  research-based  instructional 
strategies. 


All  of  the  modules  communicate  to  each  other  via  defined  protocols.  This  means  that  you  can 
easily  replace  one  version  of  a  module  with  another,  which  assists  in  setting  up  experiments  and 
training  with  a  variety  of  applications.  You  can  make  one  module  and  know  with  confidence  that 
the  rest  of  the  modules  will  do  their  duty.  Research  done  for  GIFT  can  make  its  way  back  to  the 
community. 

You  can  download  GIFT  for  free  (for  life)  and  get  support  for  your  development  at 
www.gifttutoring.org. 


IMPORTANT  THINGS  TO  REMEMBER  ABOUT  GIFT 

GIFT  is  Modular 

You  can  change  one  Domain  of  instruction  for  another... 
...And  still  have  instructionally  valid  technique 

You  can  test  one  pedagogical  theory  against  another... 
...And  maintain  experimental  control 

You  can  make  an  Engagement  Detector  for  a  Learner  Module 
Without  knowing  about  the  training  domain... 

...or  how  it  will  affect  micro-adaptation 

You  can  put  your  research  to  use. 

www.GIFTtutorinq.ora 
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Slides  30-48  discuss  the  various  releases  of  GIFT  and  their  affiliated  capabilities. 


0  RDECOM 


ARL 


GIFT 

Releases 


Well-Defin«4 

Domains 


Complax 

Domains 


Domain 

Modeling 


Interaction 

Modes 


Expert 

Modeling 


Adaptive 

Tutoring 

Research 


Authoring 


Reuse 


Strategy 

Selection 


Content 

Modeling 


Instructional 

Management 


Learner 

Modeling 


Learning 

Tutor 

Effect 

Modeling 

SFC  Paul  Ray  Smith  Simulation  &  Training  Technology  Center 


_  TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


GIFT  has  been  continuously  developing  and  growing  since  its  initial  release  in  May  2012.  The 
initial  release  included  the  fundamental  modules  and  elements  that  are  still  part  of  the  current 
releases.  An  initial  version  of  the  Module  Monitor  is  pictured  in  Slide  31. 


W  RDECOM 


GIFT  1.0 

(Baseline  Release  -  May  2012) 


ARL 


Architecture,  Toois  & 
Methods 

-  Domain-Independent  Processes 

-  Service-Oriented  Architecture 

•  Apache  Active  MQ 

-  Functionai  Moduies 

•  Pedagogical,  Learner, 
Domain,  Sensor,  Module 
Monitor,  Learning 
Management  System 
(persistent  Learner  Model) 

•  Tutor-User  Interface 
(supports  web-enabled 
content) 

-  Pedagogy  -  Variable  feedback 
based  on  performance 


SFC  Paul  Ray  Smith  Simulation  &  Training  Technology  Center 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


The  initial  release  also  included  the  ability  to  record  information  from  sensors,  such  as  EEG 
(electroencephalography),  a  temperature/humidity- sensitive  mouse,  and  a  self-assessment  sensor. 


16 


0  RDECOM 


GIFT  1.0  4 

(Baseline  Release  -  May  2012) 


ARL 


•  Sensors 

-  EEG  -  Emotiv  Epoch 

-  Temperature  and  humidity-sensitive  mouse 

-  Software-based  surrogate  sensor 

•  sensitivity  testing 


SFC  Paul  Ray  Smith  Simulation  &  Training  Technology  Center 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


Research  has  been  performed  with  the  various  releases  of  GIFT.  An  initial  experiment  was 
performed  to  assess  low-cost  sensors  and  their  impact  on  learning  cognition  and  affect.  This 
work  helped  to  inform  the  future  direction  of  sensors  within  GIFT.  Work  is  consistently  being 
done  with  GIFT  to  improve  and  add  to  its  capabilities. 


The  initial  release  included  support  for  Virtual  BattleSpace  2,  a  serious  game,  as  an  external 
training  application. 


W  RDECOM 


GIFT  1.0 

(Baseline  Release  -  May  2012) 


ARL 


Tutoring  Domains 
-  Virtuai  BattieSpace  2 

•  Training  Support  Packages 

-  lED  Report 

-  Surprise  Attack 

-  Room  Ciearing 

•  Presence  Patroi 


SFC  Paul  Ray  Smith  Simulation  &  Training  Technology  Center 


_  TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 
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0  RDECOM 


GIFT  1.0  4 

(Baseline  Release  -  May  2012) 


ARL 


•  Related  Experiments  Using  GIFT 

-  Low  Cost  Sensors  for  Real-time  Assessment 
of  Learner  Cognition  and  Affect 

•  Leveraged  VBS2  Training  Environment 

•  Emotion  Induction  Techniques  (EITs) 

•  Ciassifiers 


Cognitive  States  that 
Impact  Learning 


Attention 

Distraction 

Drowsiness 

Engagement 

Flow 

Workload 

Affective  States  that  Impact  Learning 


•  Anger/ 

•  Joy 

Frustration 

•  Motivation 

•  Boredom 

•  Sadness 

•  Confidence 

•  Shame 

•  Confusion 

•  Surprise 

•  Fear/ 

•  Wonderment/ 

Anxiety 

Awe 

SFC  Paul  Ray  Smith  Simulation  &  Training  Technology  Center 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


Additional  features  are  being  consistently  added  to  GIFT  based  on  both  research  objectives  and 
user  input.  GIFT  2.0  was  upgraded  by  including  a  number  of  different  tools  that  are  of  use  to 
course  designers  and  researchers. 


0  RDECOM 


GIFT  2.0 

(Tools  Release  -  Nov  2012) 


ARL 


•  Architecture,  Tools  &  Methods 
Added 

-  XML-based  Authoring  Tools  to 
support 

•  Surveys  and  Tests 

•  Addition/Removal  of  Sensors 

•  Changing  Learner  Models 

•  Authoring  of 
Content/Assessments 

•  Creation  of  Courses  and 
Experiments 

•  Analysis  of  Experimental 
Data 

-  Virtual  Human  Plug-In 

•  Media  Semantics  Characters 


•GIFT  Admin  Server - 

[j  LaufvhAdnwn  Server  | 

[  Open  Survey  Authoring  System  | 


SFC  Paul  Ray  Smith  Simulation  &  Training  Technology  Center 


_  TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


Additional  sensors  were  added  to  GIFT  2.0.  Once  an  interface  for  a  commercial  sensor  is  added 
to  GIFT,  the  GIFT  community  benefits  by  never  having  to  integrate  that  sensor  again. 
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0  RDECon/T^  GIFT  2.0  Jkm 

^  (Tools  Release -Nov  2012)  MI\L 


•  Sensors  Added 

-  Q-Sensor  (electro-dermal  activity, 
temperature,  acceleration) 


GIFT  2.0  also  included  the  addition  of  Microsoft  PowerPoint  as  a  training  application.  This 
addition  is  extremely  useful,  as  it  makes  it  easier  to  harness  reusable  content  in  the  form  of 
PowerPoint  presentations.  Additionally,  PowerPoint  is  a  program  that  individuals  of  many  skill 
levels  are  familiar  with,  and  it  increases  the  flexibility  that  GIFT  course  designers  have  with  their 
included  material. 


0  RDECon/rlk  GIFT  2.0  JtO! 

^  (Tools  Release- Nov  2012)  MI\L 


•  Tutoring  Domains  Added 
-  PowerPoint 

•  Tacticai  Combat  Casuaity  Care 

-  Care  under  fire  principies 

-  Hemorrhage  controi 


The  photo  in  Slide  38  demonstrates  an  experiment  in  action.  GIFT  is  visible  on  the  left  side  of 
the  screen,  with  the  training  application  on  the  right.  Additionally,  the  participant  is  wearing  the 
Q-sensor,  which  measures  electrodermal  activity. 
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GIFT  3.0  included  tools  such  as  SIMILE,  which  made  the  ability  to  link  real-time  interactions  to 
learning  objectives  more  straightforward  for  GIFT  course  authors.  In  its  present  form,  SIMILE 
works  for  TC3SimA^medic;  however,  future  work  is  being  done  that  will  allow  it  to  work  with 
additional  training  applications.  Further,  the  3.0  release  included  the  engine  for  Macro  and  Micro 
Adaptive  Pedagogy  (eM2AP)  for  managing  instruction  and  elements  of  AutoTutor  through 
dialogues  and  tutoring  scenarios. 


•  Integrates  Student  Information  Models  for  Intelligent 
Learning  Environments  (SIMILE) 

-  New  Pedagogy 

•  Domain-Independent  Macro-adaptive  Strategies  based 
on  iearner  motivation  and  expertise 

-  AutoTutor  Dialogues  and  Tutoring  scenarios 


SFC  Paul  Ray  Smith  Simulation  &  Training  Technoiogy  Center 
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GIFT  3.0  included  a  very  useful  feature:  the  ability  to  export  a  created  tutor.  This  allows  more 
flexibility  in  the  way  that  an  individual  student  will  interact  with  GIFT.  Additionally,  a  full 
installation  of  GIFT  no  longer  needs  to  be  completed  on  each  computer  that  will  be  running  the 
developed  tutor. 
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The  Human  Affect  Recording  Tool  (HART)  was  included  with  the  release  and  can  be  used  by 
researchers.  HART  is  an  app  for  the  Android  platform  that  implements  the  Baker-Rodrigo 
Observation  Method  Protocol  (BROMP)  1 .0,  a  method  for  assessing  human  affect  and 
engagement  in  field  settings,  allowing  for  synchronization  between  field  observations  and  log 
files  of  student-software  interaction. 


0  HDECOM 


GIFT  3.0  (Experiments 
Release  -  May  2013) 


ARL 


Architecture,  Tools  & 

Methods  (continued) 

-  Export  Tutor  Function 

-  Human  Affect 
Recording  Tool  (HART) 

•  Android  app  for  assessing 
human  affect  and  engagement 
in  field  settings  per  Baker- 
Rodrigo  Observation  Method 
Protocol  (BROMP) 
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HART  is  described  in  Baker  et  al.  (2012)  and  implements  the  protocol  described  in  Ocumpaugh 
et  al.  (2012). 

Additionally,  the  ability  to  track  facial  expressions,  posture,  and  head  pose  was  added  with  the 
integration  of  the  Microsoft  Kinect  as  a  sensor. 
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GIFT  3.0  (Experiments 
Release  -  May  2013) 
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•  Sensors  Added 
-  Kinect 
•  Filters 
-  Kinect 


-  MultiSense  -  Perception  Markup  Language 

»  real-time  facial  expressions,  body  posture 
»  Schrer,  etal  (2012) 

-  Generalized  Adaptive  View-based 
Appearance  Model  (GAVAM) 

»  head  pose  estimation 
»  Morency,  Whitehill  &  Movellan  (2008) 
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GIFT  3.0  also  included  additional  example  courses  that  used  TCSSinWmedic.  Additional 
researeh  was  condueted  using  GIFT  as  a  test  bed  to  examine  the  self-reference  effeet  in  eontext 
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of  cognitive  tasks  in  a  computerized  learning  environment.  The  materials  that  were  used  in  this 
study  were  released  as  an  additional  course  titled  “Logic  Puzzle  Tutorial”  in  GIFT  4.0. 
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GIFT  3.0  (Experiments 
Release  -  May  2013) 


ARL 


Related  Experiments  Using  GIFT 
'  Explicit  Feedback  with  Game-base  Training 

-  Training  to  Solve  Logic  Puzzles  (cognitive  tasks) 

-  Automated  Detection  of  Engagement  and  Affect 
Leveraging  Interaction  Patterns  and  Sensors 
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GIFT  4.0  also  included  an  enhanced  version  of  eM2AP,  additional  sensor  filters,  and  integration 
with  new  training  domains  (Marksmanship  and  Logic  Grid  Puzzles)  based  on  experimentation 
with  GIFT  3.0. 


0  HDECOn^  GIFT  4.0  (November  2013)  ARL 


•  Architecture,  Tools  &  Methods 

-  Enhanced  Learner  Affect  Models 
(interaction/sensor) 

-  Enhanced  Performance  Assessment  Engine 

•  Pedagogical  Authoring  functionality  added 

-  New  Sensor  Filters 

•  Galvanic  Skin  Response  (GSR)  data 

•  Heart  Rate  data 

-  New  Training  Domains 

•  Marksmanship 

•  Logic  Grid  Puzzles 
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As  mentioned  previously,  a  psychomotor  training  domain.  Marksmanship,  was  facilitated 
through  capabilities  in  GIFT  4.0.  This  allows  for  the  development  of  expert  marksmanship 
models.  Slide  44  displays  the  interaction  between  the  different  GIFT  modules  and  the  marksman 
(learner). 
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0  ROFCOn^  GIFT  4.0  (November  2013)  AfiL 


•  Related  Experiments  Using  GIFT 


-  Adaptive  Marksmanship  (psychomotor  tasks) 
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Further,  GIFT  4.0.  is  being  used  to  assess  future  capabilities.  As  it  continues  to  develop,  GIFT 
will  be  adapted  to  support  use  by  multiple  learners  in  a  server-based  environment.  Further,  GIFT 
is  being  integrated  into  a  mobile  learning  environment  (The  US  Army  Research  Laboratory’s 
[ARL’s]  Soldier  Centered  Army  Learning  Environment  [SCALE])  as  a  learning  engine. 
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•  Related  Experiments  using  GIFT  (continued) 

-  Scaiabiiity  &  Performance  Testing 

•  discrete  event  modeiing  of  network 
performance 

•  anaiysis  of  aiternatives 

-  e.g.,  nested  ActiveMQs 

-  GIFT  as  learning  engine  for  SCALE 

•  Soidier-Centered  Army  Learning 
Environment 

•  Vaiidation  Experiments 
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Research  efforts  are  ongoing  to  mature  architecture,  authoring  tools,  and  modeling  of  the  learner. 
As  they  mature,  these  capabilities  are  integrated  within  GIFT. 
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GIFT  2014-2  (Forecasted 
Capabilities  -  August  2014) 


ARL 


Architecture,  Tools  & 

Methods 

-  Generalized  Micro¬ 
adaptation  Strategies 

-  Automatically 
Generation  of  Expert 
Models 

-  Tutor-Game 
Integration  tools 

-  Real-time  trainee 
modeling 

-  Thin/Web  Client 
functionality 
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GIFT  2014-2  and  future  versions  are  moving  toward  separate  interfaces  and  tools  for  different 
users  (learners,  power  users,  and  domain  experts).  In  addition,  there  will  be  different 
configurations  for  different  situations/environments  (e.g.,  classrooms,  experiments,  mobile 
learning). 


0  RDECOn/Tlk  dFT  2014-2  (Forecasted  Jttff 

^  Capabilities -August  2014)  MI\L 

•  Architecture,  Tools  & 
Methods  (continued) 

-  Develop/enhance  user- 
dependent  interfaces 
•  Learners,  Domain 
Experts,  Trainers, 

1 

- 1  Us*r(s) 

[  Learn.r  | 

[  Trainar  | 

[  Davelopar  | 

[  Itos.arch.r  | 

I  Domain  Export  | 

[  Dosignor  j 

Developers, 
Instructional 
Designers,  Researchers 

-  Configurable  deployment 
(classroom,  experimental 
station,  etc.) 

-  Virtual  Human  Toolkit 
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ARL  has  the  goal  for  GIFT  to  be  a  learning  engine  for  several  types  of  training  environments. 
Targets  of  opportunity  include  the  US  Department  of  Energy’s  National  Training  &  Education 
Resource,  the  US  Advanced  Distributed  Learning  Initiative’s  Personalized  Assistant  for 
Learning  (PAL),  and  ARL’s  SCALE. 
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•  Continue  to  leverage  AutoTutor  and  AutorTutor 
Lite  functionality 

-  Speech  Act  Analysis,  Complex  Dialog  Management, 
Artificial  Intelligence  Markup  Language,  Learner 
Characteristics  Curves 

•  Enhanced  cognition  and  affect  detectors 

•  Extend  learning  environments  to  military  training 
domains  (complex  and  ill-defined  domains) 

•  GIFT  as  learning  engine  for  PAL 


(ft 
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TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


Demo  of  GIFT 
bootup 


PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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4.  Learner  Modeling 


There  are  2  key  steps  in  learner  modeling:  data  acquisition  and  state  classification.  Learner  data, 
as  noted  in  the  adaptive  tutoring  learning  effect  model,  may  be  acquired  in  real-time  from 
sensors  or  learner  input  (e.g.,  surveys,  interaction  with  training  environment)  or  in  pretutoring 
from  a  long-term  learner  model  (LTLM)  sometimes  called  a  persistent  learner  model.  Slide  51 
examines  low-cost  sensors  for  acquiring  learner  data. 


W  RDECOM 


Learner  Modeling 
Learner  Data  Acquisition 


ARL 


What  Is  It? 

•  Tools  and  methods  to  unobtrusively  acquire  learner  data  (behaviors 
and  physiological  measures)  to  support  learner  state  classification 
in  real-time  using  low-cost  ($50-200)  sensors 

What  Does  It  Offer? 

•  The  ability  to  affordably  model  learner  cognitive,  affective  and 
psychomotor  states  in  real-time  as  a  basis  for  instructional 
decisions 

Technical  Approach: 

•  Identify  a  small  set  of  low-cost  sensors  to  acquire  learner  data  to 
support  accurate  prediction  of  key  learner  states: 

•  attention,  engagement,  cognitive  load 

•  confusion,  boredom,  frustration,  anxiety,  anger 

•  motivation 


T 


ii 


EEG  ~  $200 


Kinect  ~  $100 
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Slide  52  reviews  approaches  for  using  learner  data  to  classify  learner  states  (e.g.,  emotions, 
engagement,  performance,  competence).  Of  particular  note  are  the  preliminary  findings  on  this 
slide. 
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Learner  Modeling 
State  Classification  Methods 


ARL 


What  Is  It? 

•  Tools  and  methods  to  unobtrusively  assess  learner  states  (e.g.,  engagement,  performance, 
emotions)  in  real-time  using  machine  learning  techniques 


What  Does  It  Offer? 

•  The  ability  to  accurately  model  learner  cognition,  affect  and  physical  states  in  real-time  as 
a  basis  for  tailoring  instruction 


Technical  Approach: 

•  Explore  new  machine  learning  methods  and  develop  detectors 
(classifiers)  to  accurately*  predict  key  learner  states 

•  attention,  engagement,  cognitive  load 

•  confusion,  boredom,  frustration,  anxiety,  anger 

•  motivation 

Preliminary  Findings: 

•  Generalized  classifiers  are  proving  to  be  impractical 

•  Offline  individual  classification  models  are  not  reusable 

•  Real-time  classifiers  of  affect  are  of  good  quality  (-80%  accurate) 

•  Real-time  classifiers  of  cognition  are  not  as  good  (<60%  accurate) 


‘Accuracy  =  (True  Positive  +True  Negative  Predictions)/(Totai  Predictions) 


Brawner,  K.  W.  (2013).  Modeling  Learner  Mood  In  Realtime  Through 
Biosensors  For  Intelligent  Tutoring  Improvements.  University  of  Centrai 
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The  learner  states  of  interest  during  tutoring  are  primarily  centered  in  3  areas  that  moderate 
learning  (acquisition  of  knowledge  and  skills):  cognition,  motivation,  and  affect  (see  Slide  53). 
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Learner  Modeling 
Approaches 


ARL 


*  Approach;  investigate  &  modei  significant 
infiuencers  of  teaming 


•  Cognitive  modeiing 

•  cognitive  load,  engagement  (Lepper  &  Wooiverton,  2002) 

•  attention,  distraction,  drowsiness,  engagement,  flow,  and 
workload  (Carroll,  et  ai,  2011;  Kokini,  et  ai,  2012) 

•  Motivational  modeling 

•  personality,  values,  goats,  interests  (Lepper  &  Wooiverton, 
2002) 

•  Affective  modeling 

•  confusion,  boredom,  frustration,  engagement/fiow,  curiosity, 
anxiety,  delight,  and  surprise  (Graesser  &  D’Meiio,  2012) 

•  mood  modeling  -  pleasure,  arousal,  and  dominance 
(Mehrabian,  1996;  Sottiiare  &  Proctor,  2012;  Brawner,  2013) 
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The  results  of  an  experiment  that  sampled  low-cost  sensors  and  their  ability  to  reliably  detect 
learner  states  is  shown  in  Slide  54. 
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Learner  Modeling 
Approaches  (continued) 
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Approach:  Investigate  low  cost  sensor  technologies  to  inform 
classification  of  key  infiuencers  of  learning 


•  survey  of  behavioral  &  physiological  sensors  (Carroll,  et  ai,  2011) 


Slain 

Smwr 

An^  FruMralwia  Ikwcdntn 

M<ilK<n  Deteelnr 

$100 

Anger  Fnisiraliiin.  Fear 
.Airaci>.  Borednm 

Ficjil  Rale  M<inil<ir 

$100 

.Anger  FnisIniHiii.  Btiredum 

iTuir  Pnasurc 

SenMirr 

$M0 

Stiitr.s 

Sensor 

Cost 

Engagement 

Chair  Pressure 
Sensors 

-$200 

Attention,  Engagement. 
Workload 

EEG 

$200 

Attention.  Workload 

Eye-tracker 

$500 

•  state  classification  using  physiological  sensors  (Brawner  and 
Goldberg,  2012;  Goldberg  &  Brawner,  2012;  Kokini,  et  ai,  2012) 


•  EEGs  -  Advanced  Brain  Monitoring,  Emotiv  Epoch,  and  Neurosky 

•  GSR  sensors  -  Biopac,  Affectiva  Q  sensor 

Brawner,  K.  and  Goldberg,  B.  (2012).  Realtime  Monitoring  of  EGG  and  GSR  Signals  during  Computer-Based  Training.  In 
Proceedings  of  the  Inteiligent  Tutoring  Systems  (ITS)  2012  Conference,  Chania,  Crete. 


Goldberg,  B.  &  Brawner,  K.  (2012).  Efficacy  of  Measuring  Engagement  during  Computer-Based  Training  with  Low-Cost 
Eiectroencephaiogram  (EEG)  Sensor  Outputs.  In  Proceedings  of  Human  Factors  and  Ergonomics  Society  Annual  Meeting 
(HFES201 2).  Boston,  MA. 
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Various  machine  learning  techniques  have  been  evaluated  for  use  in  classifying  learner  states. 
Slide  55  illustrates  a  clustering  technique  called  “growing  neural  gas”.  Learner  data  may  be 
labeled  (supervised),  unlabeled  (unsupervised)  or  semi-supervised.  If  you  are  interested  in 
machine  learning  techniques,  check  out  WEKA,  an  open-source  software  tool  for  machine 
learning:  http://www.cs.waikato.ac.nz/ml/weka/. 
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Learner  Modeling 
Approaches  (continued) 


ARL 


•  Approach:  clustering  and  classification  using 
supervised,  semi-supervised,  and  unsupervised 
machine  learning  techniques 


•  offline  modeling 

•  leading  to  macro- 
adaptive  instructional 
decisions 

•  online  real-time  assessment 

•  leading  to  micro- 
adaptive  instructional 
decisions 


Growing  Neural  Gas  Clustering  Technique 
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In  an  effort  to  develop  ITS  best  practices  for  learner  modeling,  ARL  convened  a  group  of  experts 
in  2012  to  examine  the  state  of  practice,  emerging  concepts,  and  future  directions.  This 
information  was  captured  in  the  first  volume  of  the  Design  Recommendations  for  ITSs,  which  is 
available  for  free  at:  https://gifttutoring.org/documents/42. 
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Learner  Modeling 
Advisory  Board 


ARL 


University  of  Memphis  -  September  2012 


•  ARL-HRED  (Sottilare,  Holden,  Goldberg,  Brawner,  Sinatra) 

•  University  of  Memphis  (Graesser,  Hu,  OIney,  Pavlik,  Rus) 

•  OSD  ADL  Initiative  (Durlach) 

•  Carnegie  Mellon  University  (Aleven) 

•  Worcester  Polytechnic  Institute  (Baker) 

•  Arizona  State  University  (VanLehn,  Burleson) 

•  Institute  for  Defense  Analyses  (Fletcher) 

•  Office  of  Naval  Research  (Perez) 

•  AFRL  (Douglass) 

•  University  of  Pittsburgh  (Lesgold) 

•  North  Carolina  State  University  (Lester) 

•  University  of  Colorado  (Nielsen) 

•  Eduworks,  Inc.  (Robson) 

•  University  of  Massachusetts  -  Amherst  (Woolf) 

•  University  of  Canterbury,  NZ  (Mitrovic) 

•  University  of  British  Columbia,  CA  (Conati) 
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If  you  were  to  build  a  learner  model,  what  kinds  of  data  would  you  want  to  capture  and  what 
states  would  you  want  to  classify/detect? 
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(T)  RDECOM 
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What  should  be  in  a 
learner  model? 

Don’t  look  ahead... 
Make  your  own  list 
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Learner  data,  states,  traits,  and  demographics  may  be  useful  in  the  ITS  decision  processes,  but 
data  are  generally  expensive  to  collect  and  maintain.  So,  we  want  to  be  selective  about  which 
data  we  choose  for  our  model. 
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Updated  Adaptive  Tutoring 
Learning  Effect  Modei 


ARL 


LTLMs  may  span  careers  or  lifetimes  of  learners  and  are  used  to  store  enduring,  variable,  and 
transient  characteristics  of  the  learner,  including  traits,  states,  and  demographics. 
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Sensor  Module 


Persistent 

Long  Term  Learner  Model: 


LRS 

LTLM 


•  Traits 

•  States 

•  Demographic  Data 
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Slide  60  shows  some  student  (learner)  data  that  is  separated  into  general  information  and 
recorded  behaviors. 


Paneva,  D.  (2006,  September).  Use  of  Ontology-based  Student  model  in  Semantic-oriented  Access  to  the  Knowledge  In  Digital 
Libraries.  In  proc.  of  HUBUSKA  Fourth  Open  Workshop  “Semantic  Web  and  Knowledge  Technologies  Applications”,  Varna, 
Bulgaria  (pp.  31-41). 


Learner  characteristics  may  be  enduring,  but  even  enduring  characteristics  may  change  over  a 
long  period  of  time.  For  this  reason,  the  label  of  long-term  learner  model  may  be  more 
appropriate  than  persistent  learner  model. 
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Enduring  Characteristics...  maybe? 


•Gender 

•Culture  First  language 

•Physical  constraints  (e.g.,  color  blind/deaf) 

•Qualifications/certifications 

•Work  History 

•Education  achievements  and  history 

•Transcripts 

•Affiiiations 

•Vaiues 

•Personaiity 
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Slide  62  lists  learner  characteristics  that  are  variable  but  fairly  persistent. 


0  ROECOM 


ARL 


Variable,  but  fairly  persistent  long  term... 

•  Second  language(s) 

•  Domain  Competencies 

•  Security  Clearance  level 

•  Inductive  Reasoning  Capacity 

•  Working  memory  Capacity 

•  Divergent  Associative  Learning  Capacity 
(ability  to  connect  new  to  prior  knowledge) 

•  Learner  contact  info;  identifier 

•  Learner  relations  (e.g.,  teacher/class  id) 

•  Learner  security  credentials  (e.g.,  password) 

•  Learner  preference  information 

•  Learner  portfolio  information 
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Slide  63  lists  transient  characteristics  that  may  be  worth  tracking/detecting. 


Transient 

•  Current  task 

•  Current  time  constraints 

•  Current  affect 

•  Current  goais 

•  Current  iocation  (geographicai) 

•  Current  time  of  day/week 

•  Current  competency  modei  (within  iearning 
experience) 

•  Current  context  (e.g.,  meeting/car,  work, 
home,  etc.) 

•  Physioiogicai  state 
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TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


We  just  discussed  what  might  be  useful  to  include  in  an  individual  learner  model.  What  should 
be  in  a  model  of  a  team  of  learners? 


What  should  be  in  a 
team  model? 


Don’t  look  ahead... 
Make  your  own  list 


PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


32 


0  RDECOM 


Adaptive  Tutoring  Learning 
Effect  Chain  for  Teams 


ARL 


^  l0mrn»rM1 
dmtm 


^  I0mrn0r02 
dmtm 


^  ImmrnmrMS 
dmtm 


jnto^ 


[  Performance  ] 


[  Cognitiv  State] 


Immrnmr  Ml 
mtmtm 

tmmrnmrM2 

atmtm 


\  AWectIv  Sf  f  I 

[  Trust  ] 

[  Communication  | 


fmmrnmr  M3 
mtmtm 


tmmm 

mtmtm 


Inmtructionml  tmmm 

mtrmtmgy  ^  Immrning  ~ 

mmlmctlon  gminm 


^  Immrnmr  MN 
dmtm 


Immrnmr  MN 
mtmtm 


informs 


Fletcher,  J.D.  and  Sottilare,  R.  (2013,  in  press).  Shared  Mental  Models  and  Intelligent  Tutoring  for  Teams.  In  Sottilare,  R.,  Hu,  X., 
Graesser,  A.  and  Holden,  H.  (Eds.)  Design  Recommendations  for  inteiligent  Tutoring  Systems:  Learner  Modeting,  Voiume  i.  Army 
Research  Laboratory. 
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0  Mode^  D^evelopment 


I 

•  structured  Literature  Review 

•  Individuai  Tutoring 

•  Team  Performance 

•  Modei  Constructs 

•  What  modeis  are  needed? 

•  What  variabies  infiuence  each  modei? 

•  What  variabies  are  observabie/unobservabie? 

•  What  is  the  effect  size  of  each  variabie? 

•  How  do  we  measure  criticai  variabies? 

•  Structurai  Equation  Modeis 

•  Modei  -  Test  -  Modei 
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In  2013,  an  extensive  review  of  the  literature  was  undertaken  relative  to  the  target  team  models 
identified  by  Sottilare  et  al.  (201 1)  and  later  described  by  Sottilare  (2013).  A  sample  of  the 
outcome  for  the  performance  model  is  shown  in  Slide  68. 


0  RDECOM 


Filtering  Methodology 
for  Literature  Review 


ARL 


Found  via  search  terms 


Coded  for  relevance 


Empirical  with  quantitative  information  which  met  our 
meta-analytic  criteria  (oj/er  10,000  effect  sizes) 


^eam  Performance^  ^ 


lu 


Meta-  Analysis 


Team  Learning 
Meta-Analysis 


Team  Satisfaction 
Meta-Analysis 


^Team  Viability^ 
Meta-Analysis 


332  cases 
examined  team 
performance 


Contain 
information  to 
caiculate  r 


1 4  cases 
examined  team 
learning  outcomes 


Contain 
information  to 
calcuiate  r 
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The  results  of  the  team  performance  model  literature  review  revealed  the  following  antecedents 
(influencers)  of  team  performance  (see  Slide  69). 
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0  RDECOM 


Examining  antecedents  to  effective 
team  performance 


ARL 


Outcome 

N2 

Ro 

Rho 

%  Variance 
accounted  for 

Overall 

183 

12910 

.18 

0.19* 

30.0% 

Communication 

48 

3367 

.23 

0.26* 

26.2% 

Coordination 

25 

1798 

.21 

0.23* 

28.8% 

Conflict 

32 

2061 

-0.08 

-0.09* 

23.3% 

Coaching/Leader 

ship 

50 

3863 

.22 

0.24* 

33.3% 

1 .  K  is  the  number  of  effect  studies  analyzed  for  this  outcome 

2.  N  is  the  number  of  individuals  evaluated  in  this  outcome 

3.  Rho  is  the  corrected  effect  size,  Ro  is  the  uncorrected  effect  size;  both  weighted  for  sample  size 
*  Statistically  significant;  confidence  interval  excludes  zero 
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The  process  of  identifying  antecedents  for  each  of  the  primary  team  models  continued,  and  these 
are  being  used  to  construct  initial  team  models  for  GIFT.  Moving  forward,  we  will  examine  the 
relationship  between  individual  and  team  modeling.  Slide  70  shows  the  interaction  between  the 
tutoring  agents,  the  training  environment,  and  the  learner.  Tutoring  agents  for  teams  may  have 
similar  functions  but  will  track  progress  toward  team  objectives. 
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You  have  seen  this  diagram  before — it  is  the  individual  tutor  loop. 


W  RDECOM 


Individual  Local  Tutor  Loop 


ARL 
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The  tutor  loop  in  Slide  72  shows  a  distributed  or  server-based  interaction  between  an  individual 
learner  and  the  tutor.  How  will  this  need  to  be  modified  for  teams? 


0  RDECOn^^  Individual  Distributed  Tutor  Loop  ARL 


Internet 


Server 
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Going  back  to  the  agent-environment-learner  interaction  for  a  moment.  Slide  73  shows 
interaction  between  a  team  of  5  learners  and  the  tutoring  agents,  a  team  of  5  learners  and  training 
environment,  and  finally,  what  is  different  in  this  diagram,  interaction  between  learners  who 
must  be  accounted  for  in  the  team  models  of  the  tutor. 
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0  RDECOM 


Team  Interactions  Between 
Learners,  Tutors  and  Environment 


ARL 
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Slide  74  shows  individual  learners  interacting  with  their  own  tutor  loops  (something  must 
moderate/manage  the  team  interaction),  interaction  between  the  learners  (communication,  trust, 
performance,  shared  mental  models  [cognition,  affect]),  and  how  competency  influences  the 
other  models. 


^  RDECOM 


Team  Distributed  Tutor  Loop 


ARL 


strategy 

recommendations 

- i - 


Server 
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Team  modeling  has  evolved  into  6  state  models  with  consideration  for  2  additional  models  for 
context  and  culture  pending  the  determination  of  their  influence  on  team  performance. 
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W  ROECOM 


Team  State  Models 


ARL 


Team  Outcome  State 
Models 

•  Performance,  Learning 

•  Viability,  Satisfaction 

Team  Competency 
State  Model 

•  Surface-Level  Composition 

•  Deep  -  Level  Composition 

Team  Cognitive  State 
Models 

•  Team  Mental  Models 

•  Transactive  Memory  Systems 

•  Situational  Awareness 

Team  Affective  State 
Models 

•  Collective  Efficacy  (Attitudes),  Cohesion  (Attitudes) 

•  Justice  (Attitudes),  Effort  (Attitudes) 

•  Cooperation  (Attitudes) 

Team  Trust  State 
Models 

•  Trust,  Psychological  Safety 

Team  Communication 
Models 

•  Coordination(  Mutual  Support,  Reflexivity,  Monitoring,  Conflict 
(Task  Conflict,  Relationship  Conflict)/Conflict  Management 

•  Leadership,  Communication,  Transition  (strategy)  and  Action 
(tactic)  processes.  Interpersonal 

Other  Elements  to 
Consider 

•  Context 

•  Culture 

technology  driven,  warfighter  focused. 

Each  of  the  team  models  is  reviewed  in  Slides  76-82. 


ROECOM 


Team  Outcome 
State  Models 


ARL 


•  Performance 

•  Learning 

•  Viability* 

•  Satisfaction* 

•  Post-exercise 


A  quantitative  or  qualitative  evaluation  of  the 

status  of  team  outcomes  during  any  portion 

of  the  team’s  performance  period. 

Outcomes  include: 

(a)  the  subjective  judgment  or  objective 
evaluation  of  team  objective 
accomplishment 

(b)  the  degree  to  which  team  members 
gained  valuable  skills  or  knowledge 

(c)  the  willingness  to  continue  to  work  with 
the  same  team  in  the  future 

(d)  how  much  the  team  enjoyed  working  with 
one  another. 


Team  outcome  model  updates  are  event-driven  and  based  on  changes  to  the  Team  Outcome  State  Models.  As  team  members 
complete  assigned  tasks  and  progress  toward  team  goals/objectives,  their  performance  and  learning  is  registered  within  the 
individual  ITS’  Team  Outcome  State  Models  and  the  ITS  then  generates  an  update  message  to  the  other  ITS  so  all  Team  Outcome 
State  Models  are  synchronized  with  individual  contributions  to  team  performance  and  learning  objectives. 
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0  RDECOM 


Team  Competency 
State  Model 


ARL 


Surface  Level 
Composition 
Deep  Levei 
Composition 


Individual  team  member  characteristics  that 
have  the  potential  to  impact  or  influence  the 
accomplishment  of  team  goals. 

Individual  knowledge,  skills,  abilities,  and 
experience  in  the  domain  are  moderators  of 
success. 

Interdependence  of  tasks  and  matching  of 
skills  and  roles  may  also  influence  the 
accomplishment  of  team  goals. 

Compatible  values  (e.g.,  culture)  and 
personality  may  also  be  moderators. 


This  model  provides  an  index  of  team  competency  based  on  a  composite  of  the  competence  levels  of  individual  team  members. 
Successful/unsuccessful  performance  influences  individual  competence  and  may  influence  team  competence.  Any  significant 
changes  in  individual  performance  of  team  tasks  are  assessed  by  the  individual  competency  state  model  to  determine  if  the  threshold 
has  been  met  to  change  individual  competency  (e.g.,  beginner,  journeyman  or  expert).  Changes  in  individual  competency  may  or 
may  not  be  of  sufficient  significance  to  affect  a  change  in  the  team  competency  state  model,  but  if  a  change  in  the  team  competency 
state  model  occurs,  a  message  is  generated  to  update  the  team  competency  state  models  of  the  other  team  members. 
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See  Fletcher  and  Sottilare  (2013)  for  additional  insight  on  Shared  Mental  Models. 


0  RDECOM 


Team  Cognitive 
State  Modeis 


ARL 


•  Team  (Shared) 
Mental  Models 

•  Transactive  Memory 
Systems 

•  Situational 
Awareness 


An  evaluation  of  the  shared  cognitive  state 
of  all  team  members.  This  evaluation  can 
include  the  degree  to  which  team 
members: 

a)  structure  knowledge  in  a  similar 
manner 

b)  understand  the  roles,  expertise,  and 
expectations  of  fellow  team  members 

c)  have  a  shared  impression  of  several 
aspects  of  the  team’s  status  (e.g.,  slack 
resources,  progression  towards  team 
objectives) 


This  state  model  Is  a  compound  model  of  the  Cognitive  State  of  all  team  members.  Cognitive  State  models  already  exist  as  part  of 
the  individual  ITS,  but  synchronization  of  this  information  with  all  the  team  member’s  ITS  is  critical  in  assessing  the  function  of  the 
team.  Depending  on  the  collaborative  task  and  the  roles  of  team  members  In  accomplishing  that  task,  the  cognitive  model  may  be 
key  in  determining  instructional  strategies.  For  example,  for  some  tasks  the  weakest  understanding  of  the  task  among  team  members 
may  indicate  the  risk  of  completing  the  task  successfully.  For  other  tasks,  only  key  team  members  may  need  to  have  higher 
understanding  of  the  task  to  reach  a  successful  outcome. 


SFC  Paul  Ray  Smith  Simulation  &  Training  Technology  Center 


_  TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


39 


0  RDECOM 


Team  Affective 
State  Models 


ARL 


•  Attitudes  (not  observable) 

•  Collective  Efficacy 

•  Cohesion 


The  general  of  feelings  of  team  members 
towards  one  another  during  team 
interaction.  This  can  include,  but  is  not 
limited  to  feelings  regarding 


•  Justice 

•  Effort 

•  Cooperation 

•  Trust  (next  slide) 


Values-Attitudes-Behaviors 


(a)  the  team’s  ability  to  accomplish  their 
goals 

(b)  the  team  emotional  sentiments  towards 
one  another 


This  state  model  is  a  compound  model  of  the  Affective  State  of  all  team  members.  Affective  State  models  already  exist  as  part  of  the 
individual  ITS,  but  distribution  of  this  information  to  all  the  team  members  ITS  is  critical  in  assessing  the  function  of  the  team.  For 
example,  if  team  performance  is  below  expectations  and  the  affective  state  of  one  or  more  team  members  is  negative,  knowledge  of 
their  state  by  other  individual  ITS  provides  the  opportunity  to  prompt  their  associated  team  members  to  take  action  (e.g., 
communicate  -  support  or  direct). 
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Team  Trust 
State  Models 


ARL 


•  Trust 

•  Credibility 

•  Psychological  Safety 
(Openness) 


The  shared  belief  that  all  team  members  will  fulfill 
their  role  responsibilities,  perform  delegated  tasks, 
and  not  attack  fellow  team  members  for 
expressing  their  opinion. 


This  state  model  is  a  compound  model  of  the  Affective  State  of  all  team  members.  Affective  State  models  already  exist  as  part  of  the 
individual  ITS,  but  distribution  of  this  information  to  all  the  team  members  ITS  is  critical  in  assessing  the  function  of  the  team.  For 
example,  if  team  performance  is  below  expectations  and  the  affective  state  of  one  or  more  team  members  is  negative,  knowledge  of 
their  state  by  other  individual  ITS  provides  the  opportunity  to  prompt  their  associated  team  members  to  take  action  (e.g., 
communicate  -  support  or  direct).  This  team  state  model  is  a  compound  model  of  the  trust  states  existing  between  team  members. 
The  trust  relationships  are  bi-direction  in  that  Team  Member  „A"  may  trust  Team  Member  „B"  more,  the  same  or  less  than  Team 
Member  „B"  trusts  Team  Member  „A".  Trust  is  influenced  by  several  factors  including  perceived  competency,  perceived  integrity, 
perceived  benevolence,  knowledge  of  the  other  team  members  (Hung,  Dennis  and  Robert,  2004)  and  perceived  benefits  of  the 
relationship  (Gujral,  DeAngelis,  Fullam  and  Barber,  2006).  Since  teams  work  toward  common  goals  where  roles  and  responsibilities 
are  distributed,  perceived  competency  is  an  essential  element  of  team  performance.  The  perception  that  other  team  members  may 
be  unable  to  perform  their  tasks  is  detrimental  to  trust  and  team  performance.  Personality  may  also  play  a  part  in  trust. 


Individuals  with  low  openness  and/or  high  neuroticism  scores  in  the  Five  Factor  Model  of  Personality  (McCrae  and  Costa,  1 994)  may 
have  developed  habits  unfavorable  to  the  development  of  trust.  Low  openness  scores  might  indicate  an  unwillingness  to  disclose 
information  while  high  neuroticism  scores  might  result  in  more  frequent  perception  of  events/interactions  as  negative.  Positive  or 
negative  emotions  can  also  influence  the  assimilation  of  information  (Linnenbrinkand  Pintrich,  2002)  and  thereby  communications, 
l^standing  and  trust. 

TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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0  RDECOM 


Team  Communications 
State  Modeis 


ARL 


•Coordination 
•Mutual  Support 
•  Reflexivity 
•Monitoring 
•Confiict 

•Task  Confiict 
•Relationship  Confiict 
•Leadership 

•Leadership  Styies 
•Shared  Leadership 
•Leadership  Behaviors 


•Communication  (frequency, 
source,  receiver(s) 
•Transition  Processes 
•Action  Processes 
•Interpersonai  Processes 
•Confiict  Management 
•Organizationai  Citizenship 
Behaviors  (OCBs) 

Observable  behaviors  between  group  members 
which  either  directly  impact  progression  towards  task 
completion  or  indirectly  facilitate  synchronization 
between  team  members. 


This  model  is  composed  of  interaction  data  between  team  members  for  the  purpose  of  observing  team  cohesion  and  task  execution. 
Providing  accurate  information  in  accordance  with  operating  procedures,  providing  communications  when  asked,  repeating 
communications  to  ensure  delivery,  sharing  information  and  acknowledging  receipt  of  information  are  all  vital  actions  observed  in 
teams  with  effective  communication  skills  (U.S.  Coast  Guard,  1998).  In  team  settings  communication  among  members  builds  holistic 
situational  awareness  and  coordinates  future  actions  to  be  carried  out.  Based  on  events  and  interactions  in  a  scenario,  team 
members  are  responsible  for  updating  one  another  in  real-time 
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0  RDEcon^^  Other  Influencing  Factors  ARL 


•  Context 

•  Task  or  Project 
Characteristics 

•  Autonomy 

•  Interdependence 

•  Familiarity 

•  Team  Tenure 

•  Organization  Type 

•  Leader  Characteristics 
(Age/Tenure) 

•  Organizational  Resources 


•  Culture 

•  Individual  Culture 

•  Team  Culture 

•  Organizational  Culture 

•  National  Culture 

•  Team  Climate 

•  Organizational  Climate 

•  Diversity 
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The  team  model  development  process  is  shown  in  Slide  83. 


0  Mode^  D^evelopment 


•  Structured  Literature  Review 

•  Individuai  Tutoring 

•  Team  Performance 

•  Modei  Constructs 

•  What  modeis  are  needed? 

•  What  variabies  infiuence  each  modei? 

•  What  variabies  are  observabie/unobservabie? 

•  What  is  the  effect  size  of  each  variabie? 

•  How  do  we  measure  criticai  variabies? 

•  Structurai  Equation  Modeis 

•  Modei  -  Test  -  Modei 
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5.  Instructional  Management 


Good  instructional  practices  are  hard  to  replicate.  Modeling  expert  human  tutors  is  a  good  place 
to  start. 


W  HDECOM 


Instructional  Management 
Approaches 


Approach:  Mode!  successes  of  expert 
human  tutors 

•  INSPIRE*  model  (Lepper,  Drake  & 
O’Donnell-Johnson,  1997) 

•  facts  about  human  tutoring  (Person  & 
Graesser,  2003) 

•  importance  of  questioning  (Dillon,  1988) 

•  relation  between  deep  reasoning 
questions  and  exam  scores  (Graesser  & 
Person,  1994) 

•  politeness  strategies  (Person,  et  aL, 
1995) 


ARL 


C 


INSPIRE  Model 


intelligent  ^ 


y 

y 


indirect 
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Dialogue-based  tutors  take  turns  interacting  with  the  learner  (see  Slide  86)  to  improve  the  quality 
of  their  answers  without  giving  them  the  answer.  Dialogue-based  tutors  guide  the  learner  and 
assess  their  understanding  of  concepts.  GIFT  has  incorporated  many  of  the  web-based  services 
used  in  AutoTutor  and  AutoTutor  Lite  to  support  dialogue-based  tutoring. 


0  RDECOM 


Instructional  Management 
Approaches  (continued) 


ARL 


Tutoring  Process  (Person,  et  al,  1995,  p.  167) 


Tutor  asks  a  question. 

1  explanations  | 

Student  answers  the  question. 

1  summaries  j 

Tutor  gives  feedback  on  the  answer. 

1  examples  | 

Tutor  and  student  coiiaborativeiy 

1"  analogies  | 

improve  the  quaiity  of  (or  embeiiish) 

hints  and  prompts 

the  answer. 

I _ _ _ } 

feedback 

Tutor  assesses  student’s 

V  J 

understanding  of  the  answer” 

1  questions  | 
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SIMILE  is  a  tool  to  link  learner  actions  in  a  simulation  or  game  to  performance  assessments  and 
ultimately  to  tutor  decisions  about  instructional  options  (e.g.,  feedback,  support,  change  in 
challenge  level). 
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Instructional  Management 
Approaches  (continued) 


ARL 


Approach;  leverage  objective-task  framework  in 
existing  assessment  engines  (e.g.,  SiMiLE) 


student  Information  Models  for 
Intelligent  Learning 
Environments  (SIMILE) 

standardized,  adaptable,  and 
generic  mechanism  for  learner 
assessment  in  simulated  training 
environments 

middleware  with  tools  for  the 
creation  of  assessment  models 
that  are  distinct  and  separate 
from  the  simulation  itself 

-  Example  Shown:  Model  rules 
for  applying  a  tourniquet  in 
the  TC3  vMedic  Trainer 


m 
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The  relationship  between  the  tutor  and  the  learner  is  critical  to  the  learner’s  engagement  and 
motivation.  The  learner  should  perceive  the  tutor  as  credible  and  supportive  for  significant 
learning  to  occur.  While  this  might  not  be  important  for  single  exposures  of  the  learner  to  the 
tutor,  over  the  long  term,  it  is  critical  for  the  learner  to  develop  a  relationship  with  the 
technology. 
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0  RDECOM 


Instructional  Management 
Approaches  (continued) 


ARL 


*  Approach:  Investigate  the  influence  of  the  iearner^s 
perception  of  the  tutor 


•  credibility  and  supportiveness  of  the  tutor  (Hoiden,  2012) 

•  learner  expectations  of  the  tutor  (Hoiden  and  Goldberg, 
2011) 

•  social  pedagogical  agents  (Kim,  Xu  and  Sharif,  2008) 

•  characteristics  of  learning  companions  (Kim,  2007;  Kim, 
Bayior,  Shen,  and  PALS  Group,  2007) 


Holden,  H.  (2012).  Understanding  the  Interaction  Behavior  of  Pedagogical  Agents’  Emotional  Support  and  Competency  on 
Learner  Outcomes  and  Agent  Perceptions.  In  Proceedings  of  the  Intelligent  Tutoring  Systems  (ITS)  2012  Conference,  Chania, 
Crete. 

Holden,  H.  and  Goldberg,  B.  (2011).  Student  Expectations  and  Tutor  Acceptance:  The  Impact  on  Student  Affect  (Mood)  and 
Future  Usage  Intentions.  International  Defense  &  Homeland  Security  Simulation  Workshop  in  Proceedings  of  the  ISM 
Conference.  Rome,  Italy,  September  2011 . 
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Another  approach  to  managing  instruction  is  to  categorize  instructional  differences  in  different 
learning  domains.  Research  is  ongoing  to  determine  generalized  instructional  management 
methods  across  and  within  learning  domains. 


^  RDECon/T^  Instructional  Management  ADfi 

— Approaches  (continued)  Alflt 


•  Approach:  Investigate  the  influence  of  learning  type  in 

selecting  effective  instructional  strategies  in  computer-based 
tutoring 


•  cognitive  learning  (Anderson  and  Krathwohi,  2000) 

•  affective  learning  (Krathwohi,  et  aL,  1964;  Goieman,  1995) 

•  psychomotor  learning  (Simpson,  1972) 

•  social  learning  (Sottiiare,  et  aL,  2011;  Seller,  2001) 

•  hybrid  learning 


Sottiiare,  R.,  Holden,  H.,  Brawner,  K.  and  Goldberg,  B.  (2011).  Challenges  and  Emerging  Concepts  in  the  Development  of 
Adaptive,  Computer-based  Tutoring  Systems  for  Team  Training.  Interservice/Industry  Training  Systems  &  Education  Conference, 
Orlando,  Florida,  December  2011. 
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Each  of  the  learning  categories  (cognitive  =  thinking,  affective  =  feeling,  and  psychomotor 
=  doing)  is  reviewed  in  Slides  90-93. 
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0  RDECon^^  Cognitive  Learning  AH.L 


•  Cognitive  ieaming  (thinking)  -  Anderson  & 
Krathwohi,  2001 

-  behaviors  indicating  increasingly  complex  and  abstract 
mental  capabilities 

-  Creating  (high):  ability  to  put  parts  together  to  form  a  new  whole 

-  Evaluating:  ability  to  judge  the  value  of  learned  material 

-  Analyzing:  ability  to  break  down  material  into  its  component  parts 

-  Applying:  ability  to  use  learned  material  in  new  situations 

-  Understanding:  ability  to  grasp  the  meaning  of  material 

-  Remembering  (low):  ability  to  recall  previously  learned  material 


Anderson,  L.  W.,  and  Krathwohl,  D.  R.  (Eds.).  (2001).  A  taxonomy  for  learning,  teaching  and  assessing:  A  revision  of  Bloom's  Taxonomy  of 
Educational  Objectives:  Complete  edition,  New  York  :  Longman. 
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0  twEcon^^  Affective  Learning  /\itL 


-  Affective  learning  (feeling)  -  Krathwohl,  Bloom 
and  Masia,  1964 

-  behaviors  indicating  emotionai  growth;  the  manner  in  which 
we  handie  emotions,  such  as  feeiings,  vaiues,  appreciation, 
enthusiasms,  motivations,  and  attitude 


-  Characterizing  (commitment  -  high);  has  a  value  system  that  controls 
their  behavior 

-  Organizing  (responsibiiity);  organizes  values  into  priorities;  comparing, 
relating  and  synthesizing  values 

-  Vaiuing  (appreciation);  the  worth  or  value  a  person  attaches  to  a 
particular  object,  phenomenon,  or  behavior 

-  Responding  (interest);  active  participation  on  the  part  of  the  learner 

-  Receiving  (awareness  -  iow);  awareness,  willingness  to  hear,  selected 
attention 


Krathwohl,  D.R.,  Bloom,  B.S.,  and  Masia,  B.B.  (1964).  Taxonomy  of  Educational  Obiectives:  Handbook  II:  Affective  Domain.  New  York:  David 

TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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•  Psychomotor  learning  (doing)  -  Simpson,  1972 

-  includes  physical  movement,  coordination,  and  the  use  of  the 
motor-skill  areas;  development  of  these  skills  requires  practice 
and  is  measured  in  terms  of  speed,  precision,  distance, 
procedures,  or  techniques  in  execution, 

-  Origination  (high):  creating  new  movement  patterns  to  fit  a  particular 
situation 

-  Adaptation:  skills  well  developed  and  can  be  modified  to  fit  special 
requirements 

-  Compiex  Overt  Response:  skillful  performance  of  complex  movements 

-  Mechanism:  learned  responses  have  become  habitual 

-  Response:  early  stages  in  learning  complex  skill;  imitation;  trial  &  error 

-  Set:  readiness  to  act 

-  Perception  (iow):  ability  to  use  sensory  cues  to  guide  motor  activity 


Simpson,  E.  (1972).  The  classification  of  educational  objectives  in  the  Dsvchomotor  domain:  The  Dsvchomotor  domain.  Voi.  3.  Washington,  DC: 
Gryphon  House. 

TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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Another  approach  to  enhanced  adaptive  instruction  is  to  implement  best  practices  based  on 
learning  theories.  Two  theories  are  Component  Display  Theory  (Merrill  et  al.  1992)  and  the  Zone 
of  Proximal  Development  (Vygotsky  1978),  which  are  described  in  Slides  94  and  95, 
respectively. 
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0  RDECOM 


Component  Display  Theory 
(Merrill,  et  al,  1992) 


ARL 


I  Praaia(ilo)  j 


•  Strategies  that  account  for  the  learner’s 
state: 

•  Gain  attention  and  motivate 

•  Adapt  to  prior  knowiedge 

•  Adapt  to  type  of  knowiedge  being  presented 

•  Adapt  to  iearner  attributes 

•  Adapt  to  the  iearner’ s  abiiity  (IQ,  EQ,  adaptability...) 
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Automated  Instruction 
Zone  of  Proximal  Development 


ARL 


What  Is  It? 

•  Tools  and  methods  to  adapt  instruction  based  on 
the  learner’s  state  (e.g.,  cognitive  and  affective) 
and  level  of  domain  competence 

What  Does  It  Offer? 

•  The  ability  to  select  real-time  instructional 
strategies  which  automatically  lower/raise  the 
training  scenario  challenge  level  and  regulate  the 
type  and  frequency  of  scaffolding  (support) 
provided  to  the  learner 

Technical  Approach: 

•  Literature  Review  to  identify  negative  learning 
states  (e.g.,  anxiety,  boredom,  frustration)  and 
associated  effective  strategies 

•  Develop  classification  rules  for  incorporation 
within  GIFT  (*eM2AP) 

•  Develop  a  Markov  Decision  Processes  to  address 
uncertainty  associated  with  learner  state 
classification 

^eM^AP  =  engine  for  Macro  &  Micro-Adaptive  Pedagogy 
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\ _ (GIFT) _ 


Generalized  Intelligent  Framework  for  Tutoring 


Adaptive  Tutoring  Methods: 

•  Learner  Modeling  &  Instructional 


Strategy/Tactic  Selection 

•  Data  Mining 

•  Machine  learning  classifiers  & 
clustering  techniques 

•  Markov  Decision  Processes 


•  Instructional  Management 


Reinforcement  Learning  In  Markov  Decision  Processes  ^ 


Component  Display  Theory 
(Merrill,  et  al,  1992) 
Domain-independent  strategies 


r 


•  Macro-adaptation 


•  Micro-adaptation 
Domain-dependent  tactics 


^  Component  Display  Theory  Implement  In  GIFT  j 
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parison  of  Human  Tutors  and  Intelligent  Tutoring  Systems.  The  Journal  of  the 
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6.  Domain  Modeling 


While  domain  modeling  is  primarily  associated  with  presentation  of  content  to  the  learner,  it  also 
includes  the  following  aspects:  modeling  of  experts  (also  known  as  an  ideal  student  model), 
which  is  used  to  assess  the  progress  of  the  learner  and  identify  errors  and  misconceptions; 
assessment  of  performance;  and  presentation  of  tactics  (actions  by  the  tutor  to  present  content, 
change  content,  provide  feedback,  or  provide  support). 

Domain  models  for  typical  ITSs  today  are  in  well-defined  domains  (math  or  physics  are  popular) 
and  are  generally  procedural  in  nature  (simple).  Very  few  tutors  cover  psychomotor  domains  and 
are  generally  desktop  and  cognitive.  ARL  is  exploring  methods  to  expand  domains  to  allow  them 
to  support  tutoring  in  a  wider  range  of  domains  and  a  broader  range  of  dimensions,  as  shown  in 
Slide  98. 
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HDECOM 


Domain  Modeling 


ARL 


What  Is  It? 

•  Tools  and  methods  to  facilitate  expansion  of 
traditional  tutoring  domains  (well-defined, 
simple,  static)  to  a  broader  spectrum  of 
military  tasks  ranging  from  well-defined  to  ill- 
defined,  from  simple  to  complex,  and  from 
static  to  dynamic 

What  Does  It  Offer? 

•  The  ability  to  support  effective  tutoring 
during  training  in  increasingly  ill-defined,  n 
complex,  and  dynamic  training  domains 

Technical  Approach: 

•  Investigate  and  identify  a  range  of  military 
training  tasks  along  three  dimensions: 
definition,  complexity  and  dynamics 

•  Discover  and  innovate  methods  to  support 
the  adaptive  tutoring  learning  effect  chain  for 
military  training  tasks  of  varying  levels  of 
definition,  complexity,  and  dynamics 


dynamic 


simple 


complex 


What  is  unique  about  eiements  of  the  adaptive  tutoring 
learning  effect  chain  as  a  function  of  definition, 
complexity  and  dynamics  of  tutoring  domains? 

•  learner  data 

•  learner  states 

•  instructional  strategies 

•  instructional  tactics 

•  learning  outcomes 
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Dynamic  interaction  modes  and  their  associated  characteristics  are  explored  in  Slides  99  and 

100. 


W  HDECOM 


Adaptive  Tutoring  Research 
Technical  Approach 


ARL 


Interaction  Mode  Environment 


Learner 

Motion 


Sensor}’  Individual 
Sensors  Interaction  /Team 


head  desktop  visual  aural  individuals 

motion,  sensors  (e.g.,  olfactory  and  netw’ork- 
posture  eye  tracker,  enabled 

changes.  head  pose  teams 

gestures  estimation) 


limited  Idnetic  indoor  in 

confined 


standing,  same  as  same  as  static  visual  aural  individuals 
crouching,  static  mode  mode  phis  olfactory,  andco- 
kneeHng,  phis  limited  motion  haptic  located 
laying  locomotion  capture  teams 


indoor  outdoor  in 

standing. 

same  as 

same  as  static 

visual  aural  individuals 

confined 

crouching, 

static  mode 

mode  phis 

olfactory,  and  co- 

instrumented 

kneeling. 

phis  full 

motion 

haptk  located 

spaces 

laying 

locomotion 

capture 

teams 

J  B 

in  the  wild 

outdoor  in 

standing. 

unrestricted 

portable 

visual  aural  individuals 

unrestricted. 

crouching, 

natural 

sensor  suites 

olfactory. 

and  CO- 

uninstrumented 

spaces 

kneeKng, 

laying 

movement 

including 

motion 

haptK: 
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Domain  Modeling 
Interaction  Modes 


ARL 


What  Is  It? 

•  Research,  tools  and  methods  to  facilitate  interaction  with  learners  and  delivery  of 
instruction  during  a  broad  spectrum  of  military  tasks  ranging  from  static  to  highly  dynamic 


What  Does  It  Offer? 

•  The  ability  to  support  effective  tutoring  during  Soldier  training  in  increasingly  dynamic 
training  modes: 

static  (desktop  training) 

J  •  limited  dynamic  (adaptive  marksmanship  training) 

I  •  enhanced  dynamic  (multi-learner  tasks  in  instrumented  spaces) 
in-the  wild  (instrumented  learners) 

Technical  Approach: 

•  Evaluate  interaction  required  to  support  each  training  mode  and 
discover  methods  to  facilitate: 

•  capture  of  learner  and  environment  data 

•  assessment  of  learner  state 

•  selection  of  optimal  instructional  strategies  and  tactics 

•  presentation  of  tactics  (e.g.,  feedback,  direction) 


Evaluation  of  interaction  is  a  necessary  precursor  to  support  tutoring  in  dynamic 
military  training  domains  (e.g.,  embedded  training,  mixed  and  augmented  reality 
training,  and  live  training) 
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An  example  of  a  more  dynamic  psychomotor  domain  is  presented  in  Slide  101. 


W  RDECOivilk  Domain  Modeling 

^  Exemplar:  Adaptive  Marksmanship 


ARL 


Weapon  Events  (trigger  pulls,  aiming  points,  shot  groupings) 
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7.  Authoring 


Authoring  may  be  divided  into  2  primary  areas:  reuse  and  automation.  Improving  methods  to  be 
able  to  use  existing  training  environments  as  adaptive  tutors  is  a  major  goal  of  our  research. 
Automated  authoring  methods  are  the  most  critical  need  in  ITS  development  today.  Tutors  are 
expensive  and  laborious  to  author,  and  they  require  sets  of  specialized  skills  to  develop  them.  A 
long-term  goal  is  to  have  any  person  with  expert  domain  knowledge  be  able  to  author  an 
effective  tutoring  system. 
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author  tutors  & 
components 


ItKiucbng 


user  models 


domain-specific 

knowledge 


instructional 

strategies 


user-tutor 

interfaces 


tutor  compiler 


Authoring  Goals  for  GIFT 

(adapted  from  Murray,  1999;  Murray,  2003) 


•  Decrease  the  effort  (time,  cost,  and/or  other  resources)  for 
authoring  and  assessing  CBTS; 

•  Decrease  the  skill  threshold  by  tailoring  tools  for  specific 
disciplines  to  author,  assess  and  employ  CBTS; 

•  Provide  tools  to  aid  the  designer/author/trainer  /researcher 
organize  their  knowledge; 

•  Support  (i.e.,  structure,  recommend,  or  enforce)  good  design 
principles  (in  pedagogy,  user  interface,  etc.); 

•  Enable  rapid  prototyping  of  CBTS  to  allow  for  rapid 
design/evaluation  cycles  of  prototype  capabilities. 

•  Employ  standards  to  support  rapid  integration  of  external 
training/tutoring  environments  (e.g.,  games)  (Sottilare  &  Gilbert, 
2011) 


Sottilare,  R.  and  Gilbert,  S.  (2011).  Considerations  for  tutoring,  cognitive  modeling,  authoring  and  interaction  design  in  serious 
games.  Authoring  Simulation  and  Game-based  Intelligent  Tutoring  workshop  at  the  Artificial  Intelligence  in  Education  Conference 
(AIED)  2011,  Auckland,  New  Zealand,  June  2011. 
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0  RDEcon^  Authoring  Approaches  ARL 


•  Approach:  Investigate  methods  and  develop  tools  for  rapid 
automated  development  of  expert  models 


•  Expert  model  development  is  an  intensive  process 

•  Exploring  automated  methods  to  author  via  text  analysis 

•  Integrate  with  tutoring  system(s)  which  make 
instructionaiiy  relevant  decisions 
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0  ROFcon^  Authoring  Approaches 


Approach:  functional  user  modeling 


standard  structures 
and  graphical  user 
interfaces  for  a 
variety  of  users 


author  tutors  & 
comporrents 


GUIs  based  on 
function  (e.g., 
researcher)  and 
functional 
competency 

•  learners 

•  subject  matter  experts 

•  instructional  system 
designers 

•  system  developers 

•  trainers 
researchers 


domain-specific 

knowledge 


instructional 

strategies 


user-tutor 

interfaces 


[irKtructlonat  system 
designer  model 


researcher  model 
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0  nDFcon^  Authoring  Approaches  AKL 


•  Approach:  learner  affect  modeling 


what  does  the  tutor 
need  to  know  about 
the  learner  to 
classify  their  affect? 

how  does  the  tutor 
get  that 
information? 

which  affective 
states  are  important 
to  recognize? 

how  does 
classification  of 
state  influence 
instructional 
decisions? 


Boredom  (23%)  Confusion  (25%)  Delight  (4%) 


Flow  (28%)  Frustration  (16%) 
Graesser  and  D’Mello  (2012) 


Surprise  (4%) 
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0  ROFcon^  Authoring  Approaches 


•  Approach:  learner  configuration  authoring  too! 


•  simple  interface  for 
authoring  learner  models 

•  tree  structure  driven  by 
XML  schema 

•  prevents  learner  model 
authoring  errors  by 
validating  against  the 
learner  model  XML 
schema 

•  provides  ability  to  validate 
learner  model  using  GIFT 
source  w/o  having  to 
launch  the  entire  GIFT 
architecture 


|^[l}g  LearnerConfiguration 
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0  RDFcon^  Authoring  Approaches^/^^ 


•  Approach:  sensor  configuration  authoring  too! 

Implemented  sensors 

•  Affective  QSensor 

*  electro-dermal  activity  (EDA) 

•  skin  temperature  and  acceleration 

•  Emotiv  EEG 

•  temperature  and  humidity  mouse  (custom) 

•  Surrogate  sensors  for  temp,  humidity  and 
assessment 


behavioral 

sensors 

physiological 

sensors 

state 

classification 

modeis 


Sensors  under  consideration 

•  NeuroSky  and  ABM  EEGs 

•  Webcam  (1Hz) 

•  Zephyr  heart  rate  monitor 

•  Sonar  distance  sensor 

•  Pressure  chair  (custom) 

•  Pupii  diameter  (custom) 

•  Design  interactive  EmoPro 
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0  ROECOn^  Passive  Sensing  -  Q  Sensor  /\liL 


Bob  playing  in  traffic  :  ) 


electro-dermal  activity 


f 


temperature 


X,  Y,  and  Z  acceleration 


Research  question:  what  is  the  minimum  set  of 
sensors  needed  to  assess  engagement,  workload, 
motivational  level  and  emotional  state? 
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W  RDECOM 


Authoring  Approaches 


Approach:  Domain  Knowledge  File  (DKF)  authoring 
too!  (DAT) 


© 


simple  interface  for 
authoring  DKFs 

tree  structure  driven  by 
XML  schema 

prevents  DKF  authoring 
errors  by  validating 
against  DKF  XML 
schema 

provides  ability  to 
validate  DKF  content 
using  GIFT  source  w/o 
having  to  launch  the 
entire  GIFT  architecture 


01351  name 

Itsp  07-GFT-0137  UXO.jtc_shakarat| 

0  |3^  startLocation 

13742143.0, 3744003.75, 3546187.01 

U  0  HJg  resources 

[]  EliJS  assessment 

0  [Ug  actions 

stateTransitions,  instructionalStrategies 

stateTransitions 


0  array  ( 1  ...unbounded)  of  stateT ransition 


J  stateTransition  performance node.nodeld=‘2‘ 


U  E  jtg  [^Pice:|performance_node  |  ▼  |performance_node.nodeld=‘2~ 
0  previous  [AtExpectation  j 

0  current  |BelowExpectation| 

[]  0  [Dg  strategyChoices 


IJUkg  stateTransition  performance_node.nodeld='3’ 


Q  0(D^  instructionalStrategies 
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0  ROFcon^  Authoring  Approaches 


•  Approach:  survey  authoring  too! 


0  RDEcon^  Authoring  Approaches  AUL 


Approach;  leverage  elements  of  existing  tutoring 
systems  (e.g.,  AutoTutor  &  AutoTutor  Lite) 


SCTipt  Information  J  Information  Delivery 
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feedback  from  Avatar 
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Great  Job,  Bob! 

Bob,  you  may  not  know  what  you  are  talking  about! 
Right  but  you  said  something  similar  before. 

You  said  that  before  and  it  was  wrong  then  too! 
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0  ROFcon^  Authoring  Approaches 


•  Approach:  game-based  tutoring 


<-  C  Otocalhost  -  - 

Gift 

•  prototype 
integration  with 
VBS2 

•  reai-time 
feedback 

•  iearner  modei 
infiuences 
chaiienge  ievei 
within  game 
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components 
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Define  data  structures  and  interface 
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components 
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PowerPoint 


Serious  Games 


Content  Authoring  Toois 


Promote  Reuse 

Automate  development  of  interfaces  with  external  training  &  authoring 
capabilities  to  support  interoperabiiity  and  reuse 
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0  RDECOM 


Automated  Authoring 

Expert  Modeling 


ARL 


What  Is  It? 

•  Tools,  and  methods  to  automate  the  development  of  expert  models  (modeling  desired 
trainee  behaviors)  for  use  by  adaptive  tutors; 

•  Expert  models,  part  of  the  GIFT  domain  module,  are  used  to  assess  learner  performance 
and  the  correctness  of  learner  actions  during  tutoring 


What  Does  It  Offer? 

•  Reduced  time,  cost,  and  skill  needed  to  develop  expert  models  for  training  domains 


Technical  Approach: 

•  Investigate  and  develop  methods  to  automate  expert  model  authoring  by  extracting  rules, 
principles,  tasks,  standards,  conditions  and  hierarchical  relationships  from  text  in  field 
manuals  and  other  text-based  data  sources  through  data  mining  techniques 


W  RDECOM 


Authoring  Reuse:  Interoperability 
with  External  Environments 
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Tools  for  Rapid  Development  of 
Expert  Models  (TRADEM) 
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^  Expert  Models  (TRADEM) 


Qick  on  any  step  in  the  workflow  to  make  changes 
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0  RDECOM 


Tools  for  Rapid  Development  of 
Expert  Models  (TRADEM) 


ARL 


0  RDECOn^^  Situated  Pedagogical  Authoring 


•  What  is  it? 

-  Providing  ITS  content  in  an  environment  that  is  as 
similar  to  the  learner’s  environment  as  possible 

-  think  WYSIWYG 

-  Hypotheses:  Situated  authoring  will  enable  authors  to. . . 

-  Gain  competence  with  the  authoring  tool  more  quickly 

-  Produce  more  complete  and  pedagogically  effective 
intelligent  tutoring  content 

(than  less  situated  authoring  tools) 
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8.  Analysis  of  Effect 


An  important  part  of  ITS  research  is  to  determine  the  best  (most  effective)  methods  of  learner 
modeling,  instructional  management  (pedagogical  strategies/recommendations),  and  domain 
modeling  (e.g.,  expert  modeling,  assessment,  and  content  and  tactics  presentation)  to  optimize 
outcomes  (e.g.,  learning,  performance,  retention,  competence). 

How  might  this  research  and  analysis  be  enabled? 


0  ROECO,^  AnaiJsTs 'xJstbed  AKL 


What  Is  It? 

•  Investigation  and  creation  of  a  prototype  design  tooi  to  support  pianning,  execution,  and 
anaiysis  of  resuits  for  comparative  evaiuations  of  adaptive  tutoring  systems,  components, 
toois,  modeis,  and  methods 

What  Does  It  Offer? 

•  The  ability  to  rapidiy  assess  the  effect  of  a  particuiar  tutoring  technoiogy  compared  to 
existing  or  emerging  techniques 

Technical  Approach: 

•  Identify  and  prototype  toois  to  conduct  anaiyses: 

•  tutor  vs.  traditionai  instructional  methods 

•  intervention  vs.  non-intervention  studies 

•  iearner  modei  comparison  studies 

•  instructionai  strategies/tactics  comparison  studies 

•  abiative  tutoring  studies 


Analyses 

Gift 


most  effective 
adaptive  tutoring 
systems,  components, 
tools,  models,  and 
methods 
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The  analysis  test  bed  within  GIFT  allows  for  scientific  evaluation  of  each  element  and 
subelements  of  a  tutoring  system  design.  The  test-bed  methodology  shown  in  Slide  130  is  based 
on  Hanks  et  al.’s  (1993)  test-bed  approach.  Effect  sizes  analyzed  using  this  test  bed  are  based  on 
Cohen’s  d  (1992). 
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Analysis 
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Domain 

Module 


•Knowledge  &  Skill 
•Performance 
•Accelerated  Learning 
•Retention 


Adapt  content,  expert  model,  tactics 
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W  ROECOM 


GIFT  available  at 
www.GIFTtutoring.org 
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Generahzod  Intelligent  Framework  for  Tutoring  (GIFT) 
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0  ROECOM 


ARL 


Thank  you  for  your  attention! 
Questions? 


SFC  Paul  Ray  Smith  Simulation  &  Training  Technology  Center 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

PUBLIC  RELEASE  -  DISTRIBUTION  UNLIMITED 


66 


Bibliography 


Anderson  JR,  Corbett  AT,  Koedinger  KR,  Pelletier  R.  Cognitive  tutors:  lessons  learned.  The 
Journal  of  the  Learning  Sciences.  1995;4:167-207. 

Anderson  LW,  Krathwohl  DR.  A  taxonomy  for  learning,  teaching  and  assessing:  a  revision  of 
bloom’s  taxonomy  of  educational  objectives:  complete  edition.  New  York  (NY):  Longman; 
2001. 

Army  Training  and  Doctrine  Command  (US).  The  United  States  Army  learning  concept  for 
2015.  Fort  Monroe  (VA):  Army  Training  and  Doctrine  Command  (US);  2011. 

Baker  RSJd,  Gowda  SM,  Wixon  M,  Kalka  J,  Wagner  AZ,  Salvi  A,  Aleven  V,  Kusbit  G, 
Ocumpaugh  J,  Rossi  L.  Sensor-free  automated  detection  of  affect  in  a  cognitive  tutor  for 
algebra.  In:  Proceedings  of  the  5*  International  Conference  on  Educational  Data  Mining; 
2012  Jun  19-21;  Chania,  Greece;  p.  126-133. 

Bloom  BS.  The  2  sigma  problem:  the  search  for  methods  of  group  instruction  as  effective  as  one- 
to-one  tutoring.  Educational  Researcher.  1984;13(6):6. 

Brawner  KW.  Modeling  learner  mood  in  realtime  through  biosensors  for  intelligent  tutoring 
improvements,  [doctoral  dissertation].  [Orlando  (EL)]:  University  of  Central  Florida;  2013. 

Brawner  K,  Goldberg  B.  Realtime  monitoring  of  ECG  and  GSR  signals  during  computer-based 
training.  In:  Proceedings  of  the  Intelligent  Tutoring  Systems  (ITS)  Conference;  2012; 

Chania,  Crete,  Greece. 

Brawner  K,  Holden  H,  Goldberg  B,  Sottilare  R.  Recommendations  for  modern  tools  to  author 
tutoring  systems.  In:  Proceedings  of  the  Interservice/Industry  Training  Systems  &  Education 
Conference;  2012  Dec;  Orlando,  FL. 

Carroll  M,  Kokini  C,  Champney  R,  Fuchs  S,  Sottilare  R,  Goldberg  B.  Assessing  trainee  states  in 
intelligent  tutoring  systems:  using  low  cost  sensors  for  affective  and  cognitive  state 
modeling.  Interservice/Industry  Training  Systems  &  Education  Conference;  2011  Dec; 
Orlando,  EL. 

Cohen  J.  Quantitative  methods  in  psychology:  a  power  primer.  Psychological  Bulletin.  American 
Psychological  Association,  Inc.  I992;II2(I):I55-I59. 

Cohen  PA,  Kulik  JA,  Kulik  CLC.  Educational  outcomes  of  tutoring:  a  meta-analysis  of  findings. 
American  Educational  Research  Journal.  1982;19:237-248. 

Dillon  TJ.  Questioning  and  teaching:  a  manual  of  practice.  New  York  (NY):  Teachers  College 
Press;  1988. 


67 


Fletcher  JD,  Sottilare  R.  Shared  mental  models  and  intelligent  tutoring  for  teams.  In:  Sottilare  R, 
Hu  X,  Graesser  A,  Holden  H,  editors.  Design  recommendations  for  intelligent  tutoring 
systems:  learner  modeling,  volume  I.  Adelphi  (MD):  Army  Research  Laboratory  (US);  2013. 

Goldberg  B.  Explicit  feedback  within  game-based  training:  examining  the  influence  of  source 
modality  effects  on  interaction,  [doctoral  dissertation].  [Orlando  (FL)]:  University  of  Central 
Florida;  2013. 

Goldberg  B,  Brawner  K.  Efficacy  of  measuring  engagement  during  computer-based  training  with 
low-cost  electroencephalogram  (EEG)  sensor  outputs.  Proceedings  of  Human  Eactors  and 
Ergonomics  Society  Annual  Meeting  (HFES2012);  2012;  Boston,  MA. 

Goldberg  B,  Brawner  KW,  Sottilare  R,  Tarr  R,  Billings  DR,  Malone  N.  Use  of  evidence-based 
strategies  to  enhance  the  extensibility  of  adaptive  tutoring  technologies.  Paper  presented  at 
the  Interservice/Industry  Training,  Simulation,  and  Education  Conference;  EITSEC.  2012; 
Orlando,  FL. 

Goleman  D.  Emotional  intelligence.  New  York  (NY):  Bantam  Books;  1995. 

Graesser  AC,  D’Mello  S.  Emotions  during  the  learning  of  difficult  material.  In:  Ross  BH,  editor. 
Psychology  of  learning  and  motivation.  New  York  (NY):  Academic  Press;  2012.  Volume  57; 
Chapter  5;  p.  183-225.  ISSN  0079-7421,  ISBN  978012-3942937,  10.1016/B978-0-12- 
394293-7.00005-4. 

Graesser  AC,  McNamara  DS.  Self-regulated  learning  in  learning  environments  with  pedagogical 
agents  that  interact  in  natural  language.  Educational  Psychologist.  2010;45:234-244. 

Graesser  AC,  Person  NK.  Question  asking  during  tutoring.  American  Educational  Research 
Journal.  1994;31:104-137. 

Hanks  S,  Pollack  ME,  Cohen  PR.  Benchmarks,  test  beds,  controlled  experimentation,  and  the 
design  of  agent  architectures.  AI  Magazine.  1993;14  (4). 

Holden  H.  Understanding  the  interaction  behavior  of  pedagogical  agents’  emotional  support  and 
competency  on  learner  outcomes  and  agent  perceptions.  In:  Proceedings  of  the  Intelligent 
Tutoring  Systems  (ITS)  2012  Conference;  2012;  Chania,  Crete,  Greece. 

Holden  H,  Goldberg  B.  Student  expectations  and  tutor  acceptance:  the  impact  on  student  affect 
(mood)  and  future  usage  intentions,  international  defense  &  homeland  security  simulation 
workshop.  In:  Proceedings  of  the  I3M  Conference.  2011  Sep;  Rome,  Italy. 

Holden  H,  Sottilare  R,  Goldberg  B,  Brawner  K.  Effective  learner  modeling  for  computer-based 
tutoring  of  cognitive  and  affective  tasks.  In:  Proceedings  of  the  Interservice/Industry 
Training  Systems  &  Education  Conference;  2012  Dec;  Orlando,  FL. 


68 


Honey  MA,  Hilton  M,  editors.  Learning  science  through  computer  games  and  simulations. 
Washington  (DC):  National  Academies  Press;  2011. 

Hunt  E,  Minstrell  J.  A  cognitive  approach  to  the  teaching  of  physics.  In:  McGilly  K,  editor. 
Classroom  Lessons.  Cambridge  (MA):  MIT  Press;  1994.  p.  51-74. 

Kim  Y.  Desirable  characteristics  of  learning  companions.  International  Journal  of  Artificial 
Intelligence  in  Education.  2007;17(04):371-388. 

Kim  Y,  Baylor  AL,  Shen  E.  PALS  Group.  Pedagogical  agents  as  learning  companions:  the 
impact  of  agent  affect  and  gender.  Journal  of  Computer-Assisted  Learning. 
2007;23(03):220-234. 

Kim  Y,  Xu  B,  Sharif  A.  Pedagogical  agents  rendering  social  context  to  an  online  learning 
environment  MathGirls.  International  Transactions  on  Systems  Science  and  Applications. 
2008;4(2):99-106. 

Kokini  C,  Carroll  M,  Ramirez -Padron  WX,  Hale  K,  Sottilare  R,  Goldberg  B.  Quantification  of 
trainee  affective  and  cognitive  state  in  real-time.  In:  Proceedings  of  Interservice/Industry 
Training,  Simulation,  and  Education  Conference  (I/ITSEC2012).  2012;  Orlando,  EL. 

Krathwohl  DR,  Bloom  BS,  Masia  BB.  Taxonomy  of  educational  objectives:  handbook  II: 
affective  domain.  New  York  (NY):  David  McKay  Co.;  1964. 

Lepper  MR,  Drake  M,  O’ Donnell- Johnson  TM.  Scaffolding  techniques  of  expert  human  tutors. 
In:  Hogan  K,  Pressley  M,  editors.  Scaffolding  student  learning:  instructional  approaches  and 
issues.  New  York  (NY):  Brookline  Books;  1997.  p.  108-144. 

Lepper  M,  Woolverton  M.  The  wisdom  of  practice:  lessons  learned  from  the  study  of  highly 
effective  tutors.  In:  Aronson  J,  editor.  Improving  academic  achievement:  impact  of 
psychological  factors  on  education.  New  York  (NY):  Academic  Press;  2002.  p.  135-158. 

Lesgold  AM,  Lajoie  S,  Bunzo  M,  Eggan  G.  Sherlock:  a  coached  practice  environment  for  an 
electronics  trouble  shooting  job.  Pittsburgh  (PA):  University  of  Pittsburgh,  Learning 
Research  and  Development  Center;  1988. 

Mehrabian  A.  Pleasure-arousal-dominance:  a  general  framework  for  describing  and  measuring 
individual  differences  in  temperament.  Current  Psychology.  1996;14:261-292. 

Merrill  D,  Reiser  B,  Ranney  M,  Trafton  J.  Effective  tutoring  techniques:  a  comparison  of  human 
tutors  and  intelligent  tutoring  systems.  The  Journal  of  the  Learning  Sciences.  1992;2(3):277- 
305. 


69 


Murray  T.  Authoring  intelligent  tutoring  systems:  an  analysis  of  the  state  of  the  art.  International 
Journal  of  Artificial  Intelligence  in  Education.  1999;10(1):98-129. 

Murray  T.  An  overview  of  intelligent  tutoring  system  authoring  tools:  updated  analysis  of  the 
state  of  the  art.  In:  Murray  T,  Blessing  S,  Ainsworth  S,  editors.  Authoring  tools  for  advanced 
technology  learning  environments.  Dordrecht  (The  Netherlands):  Kluwer  Academic 
Publishers;  2003.  p.  491-545. 

Ocumpaugh  J,  Baker  RSJd,  Rodrigo  MMT.  Baker-Rodrigo  observation  method  protocol 
(BROMP)  1.0.  Training  Manual  Ver.  1.0.  New  York  (NY):  EdLab,  Teachers  College, 
Columbia  University;  2012. 

Person  NK,  Graesser  AC.  The  Tutoring  Research  Group.  Fourteen  facts  about  human  tutoring: 
food  for  thought  for  ITS  developers.  AI-ED  2003  Workshop  Proceedings  on  Tutorial 
Dialogue  Systems:  With  a  View  Toward  the  Classroom;  2003;  Sydney,  Australia, 
p.  335-344. 

Person  NK,  Kreuz  RJ,  Zwaan  RA,  Graesser  AC.  Pragmatics  and  pedagogy:  conversational  rules 
and  politeness  strategies  may  inhibit  effective  tutoring.  Cognition  and  Instruction. 
1995;13(2):161-188. 

Rose  CP,  Jordan  PW,  Ringenberg  M,  Siler  S,  Vanlehn  K,  Weinstein  A.  Interactive  conceptual 
tutoring  in  Atlas- Andes.  In:  Moore  JD,  Redfield  C,  Johnson  WL,  editors.  Artificial 
intelligence  in  education:  AI-Ed  in  the  wired  and  wireless  future.  Washington  (DC):  lOS; 
2001.  p.  256-266. 

Simpson  E.  The  classification  of  educational  objectives  in  the  psychomotor  domain:  the 
psychomotor  domain.  Vol.  3.  Washington  (DC):  Gryphon  House;  1972. 

Soller  A.  Supporting  social  interaction  in  an  intelligent  collaborative  learning  system. 
International  Journal  of  Artificial  Intelligence  in  Education.  2001;12(l):40-62. 

Sottilare  R.  Considerations  in  the  development  of  an  ontology  for  a  generalized  intelligent 

framework  for  tutoring.  International  defense  &  homeland  security  simulation  workshop.  In: 
Proceedings  of  the  I3M  Conference;  2012  Sep;  Vienna,  Austria. 

Sottilare  R.  Characterizing  an  adaptive  tutoring  learning  effect  chain  for  individual  and  team 
tutoring.  In:  Proceedings  of  the  Interservice/Industry  Training  Systems  &  Education 
Conference;  2013  Dec;  Orlando,  EL. 

Sottilare  R.  Future  concepts  for  learner  modeling.  In:  Sottilare  R,  Hu  X,  Graesser  A,  Holden  H, 
editors.  Design  recommendations  for  intelligent  tutoring  systems:  learner  modeling,  volume 
I.  Army  Research  Laboratory  (US);  2013. 


70 


Sottilare  R,  Gilbert  S.  Considerations  for  tutoring,  cognitive  modeling,  authoring  and  interaction 
design  in  serious  games.  Authoring  Simulation  and  Game-Based  Intelligent  Tutoring 
workshop  at  the  Artificial  Intelligence  in  Education  Conference  (AIED);  2011  Jun; 

Auckland,  New  Zealand. 

Sottilare  R,  Goldberg  B.  Designing  adaptive  computer-based  tutors  to  accelerate  learning  and 
facilitate  retention.  Journal  of  Cognitive  Technology:  Contributions  of  Cognitive 
Technology  to  Accelerated  Learning  and  Expertise.  2012;17(1):  19-34. 

Sottilare  R,  Goldberg  S.  Research  gaps  for  adaptive  computer-based  tutoring  systems.  In:  Best  C, 
Galanis  G,  Kerry  J,  Sottilare  R,  editors.  Fundamental  issues  in  defence  simulation  & 
training.  London  (UK):  Ashgate  Publishing;  2013. 

Sottilare  R,  Goldberg  B,  Brawner  K,  Holden  H.  A  modular  framework  to  support  the  authoring 
and  assessment  of  adaptive  computer-based  tutoring  systems  (CBTS).  In:  Proceedings  of  the 
Interservice/Industry  Training  Systems  &  Education  Conference;  2012  Dec;  Orlando,  FL. 

Sottilare  R,  Holden  H,  Brawner  K,  Goldberg  B.  Challenges  and  emerging  concepts  in  the 

development  of  adaptive,  computer-based  tutoring  systems  for  team  training.  In:  Proceedings 
of  the  Interservice/Industry  Training  Simulation  &  Education  Conference;  201 1  Dec; 
Orlando,  FL. 

Sottilare  R,  Holden  H,  Goldberg  B,  Brawner  K.  The  generalized  intelligent  framework  for 
tutoring  (GIFT).  In:  Best  C,  Galanis  G,  Kerry  J,  Sottilare  R,  editors.  Fundamental  issues  in 
defence  simulation  &  training.  London  (UK):  Ashgate  Publishing;  2013. 

Sottilare  R,  Proctor  M.  Passively  classifying  student  mood  and  performance  within  intelligent 
tutoring  systems  (ITS).  Educational  Technology  Journal  &  Society.  2012;15(2):  101-1 14. 

VanLehn  K.  The  relative  effectiveness  of  human  tutoring,  intelligent  tutoring  systems,  and  other 
tutoring  systems.  Educational  Psychologist.  20 11;46(4):  197-221. 

VanLehn  K,  Lynch  C,  Schulze  K,  Shapiro  JA,  Shelby  R,  Taylor  L,  et  al.  The  Andes  physics 
tutoring  system:  lessons  learned.  International  Journal  of  Artificial  Intelligence  and 
Education.  2005;15(3):  147-204. 

Vygotsky  LS.  Mind  in  society:  the  development  of  higher  psychology  processes.  Cambridge 
(MA):  Harvard  University  Press;  1978. 

Woolf  BP.  A  roadmap  for  education  technology.  Arlington  (VA):  National  Science  Foundation; 
2010.  Report  No.:  0637190. 


71 


1  DEFENSE  TECHNICAL 
(PDF)  INFORMATION  CTR 

DTIC  OCA 

2  DIRECTOR 

(PDF)  US  ARMY  RESEARCH  LAB 
RDRL  CIO  LL 

IMAL  HRA  MAIL  &  RECORDS  MGMT 

1  GOVT  PRINTG  OFC 
(PDF)  AMALHOTRA 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  D 
T  DAVIS 

BLDG  5400  RM  C242 

REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRS  EA  DR  V  J  RICE 
BLDG  4011  RM217 
1750  GREELEY  RD 

FORT  SAM  HOUSTON  TX  78234-5002 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  DG  J  RUBINSTEIN 
BLDG  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  ARMC  FIELD  ELEMENT 

RDRL  HRM  CH  C  BURNS 
THIRD  AVE  BLDG  1467B  RM  336 
FORT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  AWC  FIELD  ELEMENT 

RDRL  HRM  DJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FORT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

(PDF)  RDRL  HRM  CK  J  REINHART 
10125  KINGMAN  RD  BLDG  317 
FORT  BELVOIR  VA  22060-5828 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AY  M  BARNES 
2520  HEALY  AVE 
STE  1172  BLDG  51005 
FORT  HUACHUCA  AZ  85613-7069 


1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AP  D  UNGV ARSKY 
POPE  HALL  BLDG  470 
BCBL  806  HARRISON  DR 
FORT  LEAVENWORTH  KS  66027-2302 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AT  J  CHEN 
12423  RESEARCH  PKWY 
ORLANDO  EL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AT  C  KORTENHAUS 
12350  RESEARCH  PKWY 
ORLANDO  EL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  CU  B  LUTAS-SPENCER 
6501  E  11  MILE  RD  MS  284 
BLDG200A  2ND  EL  RM2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AV  W  CULBERTSON 
91012  STATION  AVE 
FORT  HOOD  TX  76544-5073 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  DE  A  MARES 

1733  PLEASONTON  RD  BOX  3 
FORT  BLISS  TX  79916-6816 

8  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  SIMULATION  &  TRAINING 
TECHNOLOGY  CENTER 
RDRL  HRT  COL  G  LAASE 
RDRL  HRT  I  MARTINEZ 
RDRL  HRT  T  R  SOTTILARE 
RDRL  HRT  B  N  FINKELSTEIN 
RDRL  HRT  G  A  RODRIGUEZ 
RDRL  HRT  I  J  HART 
RDRL  HRT  M  C  METEVIER 
RDRL  HRT  S  B  PETTIT 
12423  RESEARCH  PARKWAY 
ORLANDO  EL  32826 


72 


1  ARMY  RSCH  LABORATORY  -  HRED 

(PDF)  HQ  USASOC 

RDRL  HRM  CN  R  SPENCER 
BLDG  E2929  DESERT  STORM  DRIVE 
FORT  BRAGG  NC  28310 

1  ARMY  G1 
(PDF)  DAPE  MR  B  KNAPP 

300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 

ABERDEEN  PROVING  GROUND 


12  DIRUSARL 
(PDF)  RDRL  HR 

L  ALLENDER 
P  FRANASZCZUK 
RDRL  HRM 

P  SAVAGE-KNEPSHIELD 
RDRL  HRM  AL 
C  PAULILLO 
RDRL  HRM  B 
J  GRYNO VICKI 
RDRL  HRM  C 
L  GARRETT 
RDRL  HRS 
J  LOCKETT 
RDRL  HRS  B 
M  LAFIANDRA 
RDRL  HRS  C 
K  MCDOWELL 
RDRL  HRS  D 
A  SCHARINE 
RDRL  HRS  E 
D  HEADLEY 
RDRL  HRT  T 
R  SOTTILARE 


73 


Intentionally  left  blank. 


74 


